TNO  Defence,  Security  and  Safety 

ONGERUBRICEERD 


Nederlandse  Organisatie  voor 
toegepast-natuurwetenschappelijk 
onderzoek  /  Netherlands  Organisation 
for  Applied  Scientific  Research 


TNO-report 

TNO-DV1  2004  A 169 


Oude  Waalsdorperweg  63 
PO  Box  96864 
2509  JG  The  Hague 
The  Netherlands 


The  role  of  IMAT  solutions  for  training 
development  at  the  Royal  Netherlands  Air  Force. 

IMAT  Follow-up  Research  Part  I 


www.tno.nl 

Phone  +31  070  374  00  00 
Fax  +31  070  328  09  61 
lnfo-DenV@tno.nl 


Date 

September  2005 

Authors) 

Mevr.  Drs.  L.R.M.A.  Jacobs 

Mevr.  Dr.  D.M.L.  Verstegen 

Mevr.  Drs.  N.H.E.  Janssen 

Classification 

Ongerubriceerd 

Affiliation 

KLu 

Project  officer 

Dr.  W.F.S.  Hylkema 

Classification  date 

29-08-2005 

Title 

Ongerubriceerd 

Managementuittreksel 

Ongerubriceerd 

Abstract 

Ongerubriceerd 

Report  text 

Ongerubriceerd 

Appendices 

Ongerubriceerd 

Contract  no 

_ 

Sponsor 

Affiliation 

KLu 

Project  name 

IMAT-2 

Project  number 

015.32801 

Copy  no 

6 

No  of  copies 

18 

No  of  pages 

1 14  (incl.  Appendices,  excl.  RDP 

No  of  appendices 

1 

DISTRIBUTION  STATEMENT  A 

Approved  for  Public  Release 
Distribution  Unlimited 


All  information  which  is  classified 
according  to  Dutch  regulations  shall  be 
treated  by  the  recipient  in  the  same  way  as 
classified  information  of  corresponding 
value  in  his  own  country.  No  part  of  this 
information  will  be  disclosed  to  any  third 
party. 

The  classification  designation 
Ongerubriceerd  is  equivalent  to 
Unclassified,  Stg.  Confidentieel  is 
equivalent  to  Confidential  and 
Stg.  Geheim  is  equivalent  to  Secret. 

All  rights  reserved.  No  part  of  this  report 
may  be  reproduced  in  any  form  by  print, 
photoprint,  microfilm  or  any  other  means 
without  the  previous  written  permission 
from  TNO. 

In  case  this  report  was  drafted  on 
instructions  from  the  Ministry  of  Defence 
the  rights  and  obligations  of  the  principal 
and  TNO  are  subject  to  the  standard 

list)  conditions  for  research  and  development 

instructions,  established  by  the  Ministry 
of  Defence  and  TNO,  if  these  conditions 
are  declared  applicable,  or  the  relevant 
agreement  concluded  between  the 
contracting  parties. 


©  2005  TNO 


Netherlands  Organisation  for  Applied 
Scientific  Research  (TNO) 


ONGERUBRICEERD 


rNO  Defensie  en  Veiligheid 

MANAGEMENTUITTREKSEL 


ONGERUBRICEERD 


2 


The  role  of  IMAT  solutions  for  training 
development  at  the  Royal  Netherlands 
Airforce. 

IMAT  Follow-up  Research  Part  I 


it  r«mo\%l  cntvu 


Spu*l  VA** 

Sotnt  abwb  Mibnai  oo«ls  wo 
the  tread  wtndi  beaonewiie*)  tke  tn 
Hn«  is  «n  *raq>Ui>W  aondrtion  and  not  < 
nmnval  Hw w«p  wkiai  ckwicn  alt 
trwd  ckakmc 


Probleemstelling 
De  KLu  heeft  van  1998  tot  2001 
deelgenomen  aan  het  Europese  IMAT 
(Integrating  Manuals  And  Training)  project 
(Esprit  29175).  Daarna  heeft  de  KLu  aan 
TNO  opdracht  gegeven  voor  IMAT 
vervolgonderzoek.  Het  Europese  IMAT 
project  heeft  een  geavanceerde  set  van 
(prototype)  tools  opgeleverd  om  technische 
handleidingen  efficient  en  effectief  te 
kunnen  hergebruiken  voor  het  ontwikkelen 
van  lesmateriaal. 

Voor  het  IMAT  vervolgonderzoek  zijn  de 
volgende  doelstellingen  geformuleerd: 

1.  Het  verkrijgen  van  inzicht  in  de 
bruikbaarheid  van  de  IMAT  tools  voor 
technische  trainingsontwikkeling  voor 
andere/nieuwe  KLu-domeinen; 

2.  Vaststellen  van  mogelijke  manieren 
waarop  het  IMAT  concept  een  bijdrage 
kan  leveren  aan  opleiding  en  training 
binnen  de  KLu; 

3.  Vaststellen  hoe  de  IMAT  metadata  zich 
verhouden  tot  de  door  de  KLu 
gehanteerde  ADL-SCORM  standaard  en 
daarop  gebaseerde  tools  ter 
ondersteuning  van  het  ontwikkelen  van 
lesmateriaal; 

4.  Opbouwen  en  verspreiden  van  IMAT 
kennis  binnen  de  KLu  en  TNO. 


ontwikkelde  modellen  zijn  gevalideerd 
door  middel  van  twee  expert  interviews. 
Om  meer  inzicht  te  krijgen  in  de 
toepasbaarheid  van  IMAT  oplossingen  voor 
(technische)  trainingsontwikkeling  bij  de 
KLu  zijn  vervolgens  diverse 
gebruiksscenario’s  ontwikkeld  en 
geevalueerd  middels  expert  reviews.  Ten 
behoeve  van  doelstelling  3  heeft  er  een 
vergelijking  plaatsgevonden  tussen  de 
IMAT  en  ADL-SCORM  metadata  en  is  een 
mogelijke  combinatie  van  IMAT  en  de  op 
ADL-SCORM  gebaseerde  tool  ter 
ondersteuning  van  het  ontwikkelen  van 
lesmateriaal  is  onderzocht.  Voor 
doelstelling  4  zijn  een  aantal  IMAT 
presentaties/colloquia  gehouden  en  is  een 
IMAT  workshop  uitgevoerd. 


Mevr.  Drs.  L.R.M.A.  Jacobs 
Mevr.  Dr.  D.M.L.  Verstegen 
Mevr.  Drs.  NH.E.  Janssen 
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Beschrijving  van  de  werkzaamheden 
Om  doelstelling  1  te  kunnen  beantwoorden 
is  een  IMAT  tools  analyse  uitgevoerd  met 
als  doel  inzicht  te  verwerven  in  specifieke 
uitdagingen  voor  de  KLu  in  het  geval 
gekozen  wordt  voor  het  in  gebruik  nemen 
van  de  tools.  Om  doelstelling  2  te  kunnen 
beantwoorden  is  een  contextanalyse 
uitgevoerd,  gericht  op  Educatieve, 
Technologische  en  Onderwijskundige  (ETO) 
factoren  binnen  de  KLu.  Op  basis  van  deze 
ETO  factoren  zijn  proces-  en 
specificatiemodellen  ontwikkeld  voor  het 
implementeren  van  het  IMAT  concept  en 
tools.  De  verzamelde  factoren  en 


Bovenstaande  werkzaamheden  zijn 
beschreven  in  twee  rapporten  (TMAT 
follow-up  research  part  I*  en  ‘part  II’).  Dit 
rapport  beschrijft  de  tools-  en  de  context 
analyse.  Rapport  II  beschrijft  de  scenario 
ontwikkeling  en  de  expert  review.  De 
resultaten  van  de  werkzaamheden  leiden 
tot  gezamenlijke  condusies  die  zijn 
opgenomen  in  beide  rapporten. 

Resultaten  en  conclusies 
Op  basis  van  de  in  dit  project  uitgevoerde 
analyse  kan  worden  geconcludeerd  dat  het 
IMAT  concept  een  waardevolle  bijdrage  kan 
leveren  aan  opleiding  en  training  binnen  de 
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KLu.  De  huidige  IMAT  tools  zijn  echter, 
ondanks  de  zeer  vergevorderde  status, 
prototype  tools  die  verder  (her)ontwikkeld 
moeten  worden  om  als  volledig 
operationele  tools  te  kunnen  worden 
ingezet.  Of  dit  zinvol  is  voor  de  KLu,  zal 
afhangen  van  keuzes  die  de  organisatie 
maakt  m.b.t.  het  ontwikkelen  van 
elektronisch  lesmateriaal,  aard  en  omvang 
van  het  trainingsdomein,  maar  ook  van 
andere  zaken  die  hierop  van  invloed  zijn, 
zoals.bijvoorbeeld  ,de  invoering  van 
elektronische  technische  manuals  en  de 
Joint  Aviation  Regulations.  Het  onderzoek 
heeft  daarnaast  geresulteerd  in  een  set  van 
randvoorwaarden/eisen  voor  IMAT 
oplossingen  enerzijds  en  de  KLu 
organisatie  anderzijds  voor  een  succesvolle 
IMAT  implementatie.  Ook  is  een 


stappenplan  opgesteld  om,  desgewenst,  de 
IMAT  tools  verder  te  ontwikkelen. 

Toepasbaarheid 

Indien  de  IMAT  tools  niet  verder  worden 
ontwikkeld,  kan  de  KLu  gebruik  maken  van 
IMAT  concepten  en  kennis  bij  andere 
opleidingstrajecten  waar  elektronisch 
lesmateriaal  wordt  (her)gebruikt. 
Bovendien  kunnen  sommige  IMAT  tools, 
zoals,  bijvoorbeeld,  de  Instructional 
Scenarios  Tool  (die  didactische 
ondersteuning  biedt  bij  het  ontwikkelen 
van  elektronisch  lesmateriaal),  ook  worden 
ingezet  als  aparte  tool.  Indien  de  KLu 
overweegt  om  de  IMAT  tools  verder  te 
ontwikkelen  kunnen  de  rapporten  van  het 
IMAT  vervolgonderzoek  als  basis  dienen 
voor  het  opstellen  van  IMAT  specificaties. 
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1.  Introduction 


This  report  is  the  first  IMAT  report  describing  the  work  and  results  of  the  MAT 
follow-up  research  conducted  by  TNO  for  the  Royal  Netherlands  Air  Force 
(RNLAF)  from  2002  until  2003. 

The  purpose  of  the  MAT  follow-up  research  is: 

•  To  gain  insight  in  the  usability  of  MAT  concepts  and  prototype  tools  for  use 
within  the  RNLAF; 

•  To  determine  how  the  MAT  approach  can  be  exploited  in  the  light  of  many 
other  relevant  training  issues  in  the  technical  training  domain; 

•  To  determine  how  the  MAT  meta-data  relate  to  the  ADL-SCORM  standard 
and  the  SCO-Generator  adopted  by  the  RNLAF; 

•  To  disperse  MAT  knowledge  further  within  the  RNLAF  and  TNO; 

This  report  (MAT  Follow-up  Research  Part  I)  describes  the  new  domain 
implementation  that  was  conducted  to  gain  further  insight  in  the  usability  issues  of 
the  latest  version  of  the  MAT  Pfinal  tools.  Next,  the  factors  influencing  the 
usability  of  MAT  concepts  and  tools  in  the  RNLAF  are  described  and  an  overview 
is  given  of  factor  and  requirements  models,  which  were  developed  to  enhance 
thinking  on  future  (MAT)  training  development  solutions  for  technical  training  in 
the  RNLAF.  Finally,  the  conclusions  and  recommendations  for  the  future  use  of  the 
MAT  concept  and  tools  within  the  RNALF  are  presented,  based  upon  the  work  as 
described  in  reports  I  and  II  of  the  MAT  follow-up  research  and  the  MAT/ ADL- 
SCORM  report  (Veerman,  et  al.,  2002). 

The  second  MAT  report  (MAT  Follow-up  Research  Part  II;  Janssen,  et  al.,  2004) 
elaborates  further  on  possible  future  solutions  that  can  be  provided  by  MAT 
concepts  and  tools  through  the  development  and  expert  review  of  various 
scenarios.  This  report  also  addresses  the  relations  and  implications  between  MAT, 
ADL-SCORM  and  the  SCO-Generator,  the  RNLAF’s  template-based  authoring 
tool  for  the  development  of  electronic  training  material.  Finally,  the  conclusions 
and  recommendations  for  the  future  use  of  the  IMA  I  concept  and  tools  within  the 
RNLAF  are  presented,  based  upon  the  work  as  described  in  reports  I  and  II  on  the 
MAT  follow-up  research  and  the  IMAT/ADL-SCORM  report. 

The  structure  of  this  report  is  as  follows. 

•  Chapter  1  gives  an  introduction  of  the  MAT  project,  tools  and  IMA  1  follow¬ 
up  research. 

•  Chapter  2  describes  the  evaluation  and  validation  of  MAT  at  GGW  De  Peel. 

•  Chapter  3  contains  an  MAT  tools  analysis  and  description  of  the  new  domain 
implementation. 

•  Chapter  4  gives  an  overview  of  factors  influencing  the  use  of  the  MAT 
concept  and  tools  in  the  RNLAF 
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•  Chapter  5  presents  the  factor  and  requirement  models  that  were  developed  and 
validated  based  upon  the  factor  study  as  described  in  chapter  4. 

•  Chapter  6  presents  conclusions  and  recommendations  on  the  possible  use  of 
IMAT  concepts  and  tools  within  the  RNLAF  organisation. 


1.1  IMAT  Project 

IMAT  is  an  acronym  for  Integrating  Manuals  and  Training.  The  aim  of  the 
European  IMAT^  research  project  (de  Hoog  et  al.,  2002),  conducted  from  1998 
until  2001,  was  the  development  of  a  set  of  tools  supporting  the  effective  and 
efficient  re-use  of  the  content  of  technical  manuals  for  instructional  purposes. 

In  many  technical  domains,  the  documentation  of  the  equipment  is  the  single  and 
main  source  to  create  training  material  for  maintenance  engineers.  However, 
technical  manuals  often  are  not  directly  useable  as  training  material  and  need 
(extensive)  revising  and  re-editing  by  training  developers.  Furthermore, 
maintenance  training  needs  constant  revising  to  keep  the  training  accurate  and  up 
to  date  with  the  many  changes  in  the  system,  their  components,  and  the  technical 
manuals. 

In  technical  domains,  training  mostly  consists  of  vocational  training  within 
companies.  Instructors  often  are  training  developers  as  well,  and  usually  have  more 
(technical)  domain  expertise  than  didactic  expertise.  Experts  in  technical  domains 
are  scarce  within  their  companies  which  often  have  to  rely  upon  them  for  difficult 
maintenance  tasks  such  as  troubleshooting,  training  novices  on  the  job,  giving 
instruction,  training  development  and  keeping  training  up  to  date.  This  means  a 
heavy  burden  on  scarce  expertise. 

The  goal  of  the  IMAT  project  is  to  provide  automated  and  structured  support  to 
make  it  possible  to  search  and  edit  fragments  of  technical  manuals  into 
instructional  material.  The  IMAT  approach  foresees  the  following  steps: 

•  technical  manuals  are  the  input; 

•  the  technical  manuals  are  automatically  fragmented; 

•  the  fragments  are  indexed  with  suitable  metadata; 

•  the  indexed  fragments  are  stored  in  a  database; 

•  users  are  supported  in  searching  the  fragments  they  need  and  retrieving  them 
from  the  database. 


^  The  IMAT  project  (ESPRIT  29715)  was  partly  funded  by  the  European  Union. 
Partners  in  the  project  were:  Intelligent  Systems  Laboratory,  University  of 
Amsterdam  (NL),  TNO  Human  Factors  and  TNO  Physics  and  Electronics 
Laboratory  (NL),  Loria  (F),  Renault  (F),  AFPA  (F),  ETRA  (E),  Tecnopolis 
CSATA  Novus  Ortus  (I),  Royal  Netherlands  Airforce  (NL),  Eurisco  (F). 
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It  is  then  possible  to  export  the  instructional  material  to  an  authoring  environment, 
for  instance  commercial  off-the-shelf  software  such  as  Word,  PowerPoint, 
Authorware,  and  such  like. 


1.2  IMAT  Tools 

The  following  tools  were  developed  in  the  IMAT  project: 

•  The  Document  Analysis  Tool  (AIDAS),  which  analyses,  fragments  and 
indexes  technical  manuals  and  other  source  material. 

•  The  Authoring  Environments  Interface  Tool,  which  allows  retrieval  of 
fragments  from  the  database  and  transferral  of  selected  fragments  to  the 
authoring  environment.  This  tool  also  enables  additional  indexing  and 
annotation  of  the  fragments. 

•  The  Organisational  Memory  Tool,  which  facilitates  additional  (manual) 
indexing  and  annotation  of  the  fragments  enabling  organisational  memory  and 
learning  on  the  use  of  instructional  material  in  an  organisation. 

•  The  Ontology  Development  Tool  which  can  be  used  to  create  ontologies  in  a 
structured  way. 

•  The  Instructional  Scenarios  Tool  which  supports  the  creation  of  frameworks 
for  specific  lessons  for  maintenance  tasks.  Within  IMAT  a  lesson  framework  is 
made  visible  in  a  so-called  Instructional  Bag,  a  directory  structure  that  reflects 
the  lesson  framework.  The  framework  reflected  in  the  Instructional  Bag  can  be 
used  as  guidance  for  database  queries  and  users  can  fill  an  Instructional  Bag 
with  the  fragments  required  for  different  parts  of  the  lesson.  The  Instructional 
Bag  can  then  be  exported  to  the  Authoring  Environments  Interface  Tool. 
Alternatively,  users  can  directly  copy  and  paste  fragments  to  an  Authoring 
Environment. 

•  The  Database  Facilities  Tool  enables  IMAT  users  to  manage  the  database 
where  fragments  of  the  source  material  are  stored.  This  tool  facilitates  version 
management  of  fragments,  necessary  due  to  fragment  updates  and  various 
lifestages  of  a  fragment,  e.g.  the  transformation  of  a  fragment  to  an 
instructional  fragment. 

•  IMAT  Database:  A  Jasmine®  database  is  used  as  central  database  in  IMAT  for 
the  storage  of  fragments.  This  is  an  object-oriented  relational  database. 
Jasmine®  was,  amongst  other  factors,  chosen  for  its  ability  to  handle 
multimedia  formats. 

At  the  core  of  the  IMAT  tools  lies  a  set  of  ontologies  that  were  developed  to 
describe  fragments,  both  domain-specific  and  generic,  and  also  from  an 
instructional  viewpoint.  An  ontology  can  be  described  as  a  structured  list  of 
concepts,  keywords  and  format  descriptions.  Such  ontology  can  be  described,  e.g. 
in  a  conceptual  modelling  language  (CML)  and  be  used  by  software  tools  to 
identify  and  describe  fragments,  such  as  audio  or  text,  make  distinctions  between 
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different  types  of  fragments  such  as  titles  and  sections,  and  “recognise”  specific 
content. 

The  next  figure  gives  an  overview  of  the  general  workflow  in  the  IMAT  tools. 


The  transparent  boxes  depict  the  processes  in  the  IMAT  workflow.  The  red  arrows 
indicate  the  possible  direction  of  this  workflow  in  a  training  material  development 
process  with  IMAT  tools,  from  the  delivery  of  source  material  to  the  export  of 
fragments  to  an  external-authoring  environment.  The  orange  boxes  represent  the 
IMAT  tools,  and  the  orange  circles  represent  the  fragmentation,  description, 
domain  and  instructional  ontologies.  The  blue  boxes  represent  the  source  material 
in  various  life  stages  and  forms,  e.g.  technical  manuals,  (interactive)  electronic 
technical  manuals  and  fragments. 

Although  it  is  possible  to  use  the  whole  set  of  IMAT  tools,  it  certainly  is  not 
necessary.  One  can  for  instance  only  use  the  fragmentation  and  search  options 
provided  with  the  AIDAS  tool  or  only  the  Scenario  Wizard  of  the  Instructional 
Scenarios  Tool  as  guidance  to  create  maintenance  lessons.  A  more  detailed 
description  of  the  IMAT  tools  will  be  given  in  chapter  3:  Tools  Analysis. 
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1.3  IMAT  Follow-up  Research 

In  the  Netherlands  TNO^  and  the  Royal  Netherlands  Air  Force  (RNLAF) 
continued  the  IMAT  research  in  IMAT  follow-up  research.  The  main  goal  of  the 
IMAT  follow-up  project  is  to  get  insight  in  the  usability  of  IMAT  concepts  and 
tools  for  technical  training  development  within  the  RNLAF.  A  second  goal  is  to 
determine  how  the  IMAT  approach  can  be  exploited  in  the  light  of  many  other 
relevant  training  issues  in  the  RNLAF’s  technical  training  domain.  The  RNLAF, 
for  instance,  decided  to  follow  the  ADL  SCORM  guidelines  for  the  standardisation 
of  e-leaming  content.  Therefore,  it  is  crucial  to  know  whether  or  not  the  IMAT 
tools  are  (or  can  be)  SCORM  compliant.  See  for  more  information  on  this  part  of 
the  research:  Veerman,  et  al.,  (2002)  and  Janssen,  et  al.,  (2004).  Furthermore, 
simulation-based  training  methods  are  used  in  the  technical  training  domain.  Here 
also  technical  manuals  constitute  the  basis  of  the  training  material.  The  content  of 
technical  manuals  changes  and  is  updated  regularly.  This  means  that  for 
simulation-based  training  it  is  essential  to  track  these  changes  and  update  the 
training  material  in  an  easy  way.  Nowadays,  it  is  almost  impossible  for  the  RNLAF 
to  keep  the  training  material  of  the  simulations  up  to  date  with  changes  in  the 
technical  documentation.  The  IMAT  tools  could  assist  in  this  process.  Another 
interesting  phenomenon  in  the  RNLAF  is  the  implementation  of  Interactive 
Electronic  Technical  Manuals  (IETMs)  for  aircraft  maintenance. 

Training  and  Education  in  the  RNLAF 

Within  the  RNLAF  training  and  education  play  a  crucial  role  in  achieving  and 
maintaining  a  high  state  of  military  operations.  Technical  training  in  the  RNLAF 
often  involves  the  transfer  of  crucial  and  critical  knowledge  and  skills,  for  instance 
for  air  traffic  control,  aircraft  maintenance  and  (tactical)  fighter  control.  The  use  of 
advanced  high-tech  training  solutions  such  as  simulation  is  far  more  common  than 
in  the  civil  world.  Another  important  characteristic  of  technical  training  is  that  the 
training  material  has  to  be  updated  frequently  due  to  changes  in  the  operational 
equipment  and  technical  manuals.  For  reasons  of  personal  and  system  safety  it  is 
essential  to  work  with  up  to  date  source  material,  both  in  maintenance  and 
technical  training. 

In  the  current  situation  for  the  training  of  maintenance  engineers  the  RNLAF  is 
confronted  with  the  situation  sketched  in  section  1.1:  training  material  often  is 
developed  by  the  instructors  themselves  with  the  technical  manuals  as  most 
important  input.  Instructors  are  domain  experts  with  some  experience  in  teaching, 
but  not  much  background  in  instructional  design  and  development  of  training 
material.  In  practice,  developing  training  material  takes  up  a  lot  of  instructor  time 
especially  since  there  is  a  frequent  update  process,  e.g.  when  personnel  changes  or 
the  technical  manuals  are  updated. 
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Follow-up  Research 

During  the  European  IMAT  project,  one  of  the  RNLAF’ s  airbases,  GGW  De  Peel, 
co-operated  in  the  project  as  a  user-partner.  Considering  the  close  involvement  of 
this  user-partner  in  the  tool  development  process  (the  tools  were  specified 
according  to  their  needs),  it  would  have  been  interesting  to  focus  the  follow-up 
research  project  on  the  integration  of  the  tools  at  the  GWW  De  Peel  airbase. 
However,  due  to  reorganisations  GGW  De  Peel  could  not  continue  to  use  the 
IMAT  tools  and  be  part  of  the  follow-up  research  project.  In  chapter  2  we  briefly 
describe  these  developments  and  summarise  the  main  conclusions  of  the  last 
evaluation  at  GGW  De  Peel. 

To  address  the  issues  that  remained  open  for  the  RNLAF,  the  IMAT  follow-up 
project  was  set  up  with  the  next  research  questions  in  mind: 

a)  How  usable  are  the  IMAT  tools  for  new  domains? 

b)  What  are  the  possibilities  for  using  IMAT  knowledge  and  tools  elsewhere 
within  the  RNLAF?  Which  other  training  issues  are  relevant  in  this  context? 

c)  How  do  the  IMAT  meta-data  relate  to  the  ADL  SCORM  standard  that  the 
RNLAF  has  decided  to  adopt? 

d)  How  do  the  IMAT  knowledge  and  tools  relate  to  the  SCO-generator  for  the 
RNLAF? 

The  first  report,  IMAT  Follow-up  Research  Part  I,  addresses  question  A  and  B.  The 
second  report,  IMAT  Follow-up  Research  Part  II,  elaborates  on  question  B  and 
answers  question  C  and  question  D. 

To  answer  question  A,  we  started  with  a  summary  of  the  most  important  lessons 
learned  on  the  usability  of  the  IMAT  tools  at  GGW  De  Peel  in  the  European  IMAT 
Project  (See:  Chapter  2).  Also,  we  tried  to  implement  a  new  domain  in  the  IMAT 
tools  (See:  Chapter  3).  To  answer  question  B,  we  investigated  educational, 
technology  and  organisational  factors  influencing  the  possible  use  of  IMAT 
concepts  and  tools  in  the  RNLAF  (See:  Chapter  4)  and  developed  factor  and 
requirement  models  to  study  the  way  these  factors  interact  and  influence  the  use  of 
IMAT  in  the  RNLAF  (See:  Chapter  5).  The  models  were  validated  and  elaborated 
by  expert  reviews  and  interviewing  key  (RNLAF  and  TNO)  experts.  The 
conclusions  and  recommendations  for  the  future  use  of  the  IMAT  concept  and  tools 
within  the  RNLAF  are  presented  based  upon  the  work  as  described  in  reports  I  and 
II  on  the  IMAT  follow-up  research  in  Chapter  6. 
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2.  Evaluation  and  Validation  of  IMAT  at  GGW  de  Peel 


Section  2. 1  describes  organisational  developments  at  GGW  De  Peel.  In  section  2.2 
we  summarise  the  most  important  results  of  the  evaluations  executed  within  the 
European  IMAT  project  at  GGW  De  Peel. 


2.1  Organisational  developments  at  GGW  De  Peel  in  2002 

GGW  De  Peel  has  been  directly  involved  in  the  European  IMAT  Project 
(Integrating  Manuals  and  Training,  Esprit  Project  number  29175)  in  the  role  of 
user.  This  has  meant  that  instructors  from  GGW  De  Peel  have  collaborated  in  the 
definition  of  user  needs  for  the  methods  and  tools  to  be  developed,  as  well  as  in  the 
evaluation  of  three  IMAT  prototypes. 

2.1.1  HAWK  issues 

In  the  first  part  of  the  European  IMAT  project  the  domain  that  was  used  was  the 
training  of  maintenance  engineers  for  the  Flycatcher  (prototypes  0  and  1).  Since  the 
RNLAF  decided  not  to  continue  the  Flycatcher  maintenance  training,  the  HAWK 
domain  was  used  in  the  second  half  of  the  project  (prototype  2). 

For  the  IMAT  follow-up  project  the  following  activities  were  originally  foreseen  at 
GGW  De  Peel  in  2002: 

•  introduce  new  HAWK  material  into  the  prototype  tools  when  it  becomes 
electronically  available  (during  the  European  project  only  three  manuals  were 
available  electronically); 

•  develop  lesson  material  for  the  HAWK  course  using  the  prototype  tools 
(including  the  instructional  scenarios  tool); 

•  further  evaluation  and  validation  of  the  IMAT  concepts  and  tools  for  GGW  De 
Peel  specifically,  and  for  the  RNLAF  in  general. 

In  December  2001  the  final  versions  of  the  IMAT  prototype  tools,  with 
improvements  implemented  after  prototype  version  2,  have  been  installed  at  GGW 
De  Peel.  There  were  still  some  technical  problems  with  the  HAWK  electronic 
material.  In  the  first  few  months  of  2002  TNO  and  GGW  De  Peel  collaborated  with 
the  University  of  Amsterdam  (responsible  for  the  implementation  of  the  AIDAS 
tools)  to  solve  these  problems.  The  expectation  was  that  the  other  HAWK  manuals 
would  soon  become  electronically  available,  but  that  did  not  happen.  We  could, 
therefore,  not  introduce  new  HAWK  material  in  the  IMAT  prototype  tools. 

Around  the  same  time  it  became  clear  that  there  would  be  no  maintenance  engineer 
courses  for  the  HAWK  in  2002.  For  the  instructors  at  GGW  De  Peel  the 
development  of  lesson  material  for  the  HAWK  course  was,  therefore,  no  longer  a 
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priority.  By  September  2002,  it  became  apparent  that  the  RNLAF  would  not 
continue  training  for  new  maintenance  engineers  for  the  HAWK  as  the  system 
would  not  be  used  much  longer.  In  the  future,  the  RNLAF  plans  to  purchase  the 
PATRIOT  as  new  air  defence  system.  The  training  of  maintenance  engineers  for 
the  PATRIOT  will  probably  be  outsourced. 

2.1.2  Conclusions 

The  source  material  for  the  HAWK  was  problematic  because  of  the  low  quality  of 
the  material  (not  originally  electronic,  but  scanned  in)  and  the  layout  of  the 
documents  was  inconsistent.  Since  the  RNLAF  has  decided  to  stop  training  HAWK 
maintenance  engineers,  it  does  not  seem  sensible  to  dedicate  more  time  and  effort 
to  this  domain.  Further  evaluation  and  validation  of  the  1M  AT  concepts  and  tools 
was  not  possible  at  GGW  De  Peel,  because  no  lesson  material  for  the  HAWK  had 
been  made.  We  therefore  decided  to  try  to  implement  a  (realistic  but  not  too  big) 
domain  ourselves  in  order  to  get  a  better  view  on  how  usable  the  IMAT  tools 
would  be  for  new  domains  (see  chapter  3). 


2.2  Main  results  of  the  evaluations  within  the  IMAT  project 

In  this  section  we  summarise  the  most  important  results  of  the  evaluations  executed 
within  the  European  IMAT  project.  For  a  more  elaborate  report  see:  Verstegen, 
Veldhuis,  Staalstra,  and  Hendriks,  (2001)  and  Verstegen  et  al.  (2001). 

2.2.1  Evaluation  of  the  IMAT  concepts  and  tools:  general  conclusions 

The  evaluation  of  the  IMAT  concepts  and  prototype  tools  was  an  important  part  of 
the  European  IMAT  project.  Evaluations  took  place  at  the  three  user  sites  involved 
in  the  project: 

•  in  the  Netherlands  at  the  RNLAF,  GGW  De  Peel.  Domain:  training  of 
Flycatcher/Hawk  maintenance  personnel; 

•  in  France  at  AFP  A.  Domain:  lower  vocational  training  for  mechanics  for  the 
Renault  CLIO);  and 

•  in  Spain  at  ETRA.  Domain:  training  for  maintenance  and  repair  of  traffic 
control  systems. 

Three  domains  were  implemented  in  the  IMAT  tools  and  the  tools  were  used  to 
create  lesson  material.  Further  evaluation  data  consists  of  answers  to  questionnaires 
and  semi-structured  interviews  (Verstegen  et  al.,  2001). 

At  the  end  of  the  project  IMAT  users  saw  many  potential  benefits  of  using  the 
IMAT  concepts  and  tools: 

•  it  reduced  the  time  spent  on  making  lesson  material  and  therefore  the  costs;  and 
it  supported, 
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•  the  standardisation  of  lesson  material; 

•  the  reuse  of  source  and  lesson  material; 

•  knowledge  sharing  between  instructors  (and  other  staff);  and 

•  knowledge  management. 

The  most  important  remaining  problems  concerned  the  analysis  and  indexing  of 
source  material.  Most  of  these  problems  seemed  to  be  of  technical  nature. 

The  next  most  pressing  problem  was  the  updating  problem:  at  this  moment  the 
database  has  to  be  refilled  when  there  is  an  update  of  the  technical  manuals.  This 
means  that  all  instructor  changes  (revisions  in  fragments,  additional  fragments, 
additional  indexes,  annotations  etc.)  are  lost  and  therefore  severely  hamper  the  use 
of  the  facilities  to  revise  and  add  fragments,  add  annotations  and  add/change 
indexes. 

The  user  evaluation  confirmed  the  choice  made  earlier  in  the  IM  AT  project  to  keep 
the  different  tools  independent  and  separate  so  that  users  could  choose  which  tool 
they  wanted  to  use  and  when.  All  tools  are  connected  through  the  database. 

2.2.2  More  specific  results  from  RNLAF  users 

The  RNLAF  users  are  positive  about  the  Instructional  Scenarios  Tool.  They  could 
follow  the  lines  of  reasoning  in  scenario  construction  without  problems.  The 
resulting  list  of  Instructional  Activities  is  similar  to  the  standard  lesson  structure  as 
they  were  taught  during  their  instructor  training  (only  they  use  different  definitions, 
i.e.  introduction;  presentation;  feedback).  Making  a  lesson  structure  (i.e.  an 
Instructional  Bag)  without  the  Instructional  Scenarios  Tool  was  considered  to  be 
less  easy. 

The  RNLAF  users  found  inserting  fragments  into  an  existing  Instructional  Bag  an 
easy  way  to  make  a  lesson,  but  it  would  have  been  easier  if  drag  and  drop-functions 
were  available.  We  discovered  that  it  was  not  possible  to  insert  cut-out  parts  of 
pictures  into  the  bag-*.  Exporting  bags  appeared  easy:  the  tool  constructs  a  structure 
in  which  pictures,  texts,  video,  etc.  can  be  saved.  Cutting  and  pasting  fragments 
directly  from  the  Authoring  tool  to  PowerPoint  also  worked  without  problems. 
According  to  the  instructors  using  Instructional  Bags  was  more  useful  than 
constructing  lesson  materials  directly  with  copy  and  paste  to  the  authoring  tool. 
When  using  the  tool  in  practice,  they  would  expect  to  make  training  material  using 
Instructional  Bags  in  most  cases. 

Regarding  the  annotation  facilities,  RNLAF  users  remarked  that  the  number  of 
choices  was  rather  large.  Most  of  the  choices  apply  to  (parts  of)  lessons  and  not  to 
individual  fragments,  for  example:  misunderstandings  and  recovery  strategies  that 
can  only  be  filled  in  during  training. 
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The  instructors  were  very  positive  about  the  possibility  to  save  annotations  with 
(parts  of)  Instructional  Bags.  They  did  not  expect  to  use  annotations  with 
individual  fragments.  A  main  problem  that  occurs  when  the  database  is  updated  is 
that  the  annotations  with  fragments  could  get  lost.  It  is  not  clear  whether 
Instructional  Bags  plus  the  related  annotations  might  also  get  lost.  Ideally,  there 
should  be  a  way  of  keeping  track  where  fragments  are  used,  so  users  are  warned 
when  a  fragment  is  updated. 

2.2.3  Conclusions 

Using  the  Instructional  Scenarios  Tool  and  the  annotation  facilities  has  several 
advantages: 

•  Lesson  material  will  be  more  consistent  &  lesson  material  can  be  reused; 

•  Multiple  instructors  can  easily  use  the  same  lesson  material,  instead  of  making 
their  own  lesson  material  on  overhead  sheets  as  was  done  before.  As  a  result, 
instruction  differed  strongly  between  instructors,  and  there  was  little  (or  no) 
awareness  of  what  learning  goals  were  being  addressed.  Using  the  same 
scenario  and  material  will  ensure  consistent  lessons.  Using  the  annotation 
facilities  will  make  it  possible  to  share  and  store  experiences  and  comments. 

•  Saving  the  selected  fragments  in  the  structure  of  the  Instructional  Bag  will 
make  it  easier  to  co-operate  with  other  instructors  during  the  creation  of  lesson 
material. 

•  Using  Instructional  Bags  will  be  especially  useful  when  the  lesson  material  is 
implemented  by  other  people  (e.g.  in  Authorware  by  Multimedia  developers) 
or  when  implementation  is  outsourced. 

•  Using  Instructional  Bags  will  make  it  easier  to  revise  training  material  after 
updates,  especially  when  instructors  use  an  authoring  tool  where  fragments  are 
imported  from  file  on  the  fly.  In  this  case  out-of-date  fragments  could  be 
simply  replaced  in  the  Instructional  Bag  and  repeat  creation  of  the  lesson 
material  would  not  be  necessary. 

The  RNLAF  users  did  not  find  the  facilities  to  add/change  indexes  of  fragments, 
easy  to  use.  They  also  expected  that  it  would  be  used  less  frequently  when  the 
amount  of  material  in  the  database  was  very  large.  They  agreed  that  it  is  necessary 
to  be  able  at  least  to  change  indexes  that  are  obviously  wrong.  But  in  this  case,  the 
updating  problem  should  be  solved  as  well:  it  would  not  be  sensible  to  improve  the 
indexes  of  fragments  in  the  database,  if  these  changes  all  get  lost  when  the  database 
has  to  be  refilled  after  a  manual  update.  The  same  problems  occur  when  fragments 
are  revised  or  new  fragments  are  added  to  the  database.  Use  of  the  instructional 
indexes  was  not  foreseen  at  GGW  De  Peel. 
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3.  IMAT  Analysis 


Section  3.1  starts  with  a  description  of  the  IMAT  tools  for  a  better  understanding  of 
the  various  tools  and  the  workflow  in  the  IMAT  process.  In  section  3.2,  the  results 
of  the  tools  analysis,  based  upon  the  work  for  a  new  domain  implementation,  are 
presented. 


3.1  Overview  of  IMAT  Tools 

In  this  section  a  more  detailed  description  of  the  IMAT  tools  is  given.  The 
following  tools  are  described:  the  document  analysis  tool  (AIDAS),  the  authoring 
environment  interface  tool,  the  instructional  scenarios  tool,  the  organisational 
memory  tool,  the  database  facilities  tool  and  some  tools  which  are  integrated 
within  one  of  these  tools,  such  as  fragment  retrieval  tools  and  the  ontology 
development  support  tool. 

3.1.1  AIDAS 

AIDAS  is  an  acronym  for  the  Amsterdam  Intelligent  Document  Analysis  System 
and  is  used  in  IMAT  as  name  for  the  (automatic)  documentation  analysis  tool.  Its 
main  purpose  is  to  break  up  the  content  of  technical  manuals  into  smaller  text  and 
image  fragments.  These  are  then  provided  with  an  index,  which  describe  fragments 
from  various  viewpoints,  and  are  necessary  for  proper  fragment  identification. 
Finally  the  AIDAS  tool  enables  the  export  of  fragments  to  the  Jasmine®  database, 
which  is  used  in  IMAT  as  central  database. 

3.1. 1.1  Ontologies 

In  IMAT,  the  proposed  solution  on  problems  with  information  retrieval  and 
indexing  is  based  upon  ontology  indexing.  IMAT  uses  different  ontologies, 
representing  various  point  of  views  to  create  fragments.  These  ontologies  play  a 
key  role  in  the  IMAT  solution.  By  defining  the  ontologies,  IMAT  tools  are  able  to 
fragment,  retrieve  and  describe  content.  The  ontologies  were  developed  with  a 
close  look  at  the  work  done  in  similar  projects  on  ADL  SCORM,  LOM  and 
Ariadne  (www.adl.org)  for  the  standardisation  of  metadata  for  electronic  content. 

In  IMAT  a  distinction  is  made  between  generic  and  domain-specific  ontologies. 
The  following  ontologies  were  developed  in  the  IMAT  project  (Anjewierden  & 
Kabel,  2001): 

•  Fragment  ontology:  general  and  syntactic  viewpoint; 

•  Description  ontology:  semantic  viewpoint; 

•  Instructional  ontology:  instructional  viewpoint; 

•  Domain  ontology:  domain  specific  viewpoint. 
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The  various  ontologies  enable  the  indexing  of  fragments  from  several  points  of 
views.  The  fragment  ontology  captures  fragment  aspects  from  a  general  and 
syntactical  viewpoint.  This  ontology  enables  the  description  of  fragments  in  terms 
of  this  fragment  “is  a  video”,  or  “the  size  of  this  text  fragment  is”. 

The  description  ontology  describes  the  fragment  type  and  gives  a  semantic 
viewpoint  on  a  fragment.  By  the  use  of  this  ontology  either  the  content  of  a 
fragment  is  (briefly)  described,  or  the  location  or  structure  of  a  fragment,  such  as 
location  “TM-1234-89a”. 

The  instructional  (roles)  ontology  represents  the  instructional  viewpoint  and 
enables  the  description  of  possible  instructional  roles  for  a  fragment.  A  fragment, 
which  is  indexed  as  “video”  could  become  in  an  instructional  context  an 
“example”  or  part  of  an  “exercise”.  The  components  by  which  this  ontology  gives 
an  instructional  description  of  fragments  are  learning  goal,  knowledge  type, 
instructional  strategy,  instructional  activity,  instructor  and  learning  actions,  and 
material  use.  Parts  of  this  ontology  are  used  to  add  instructional  metadata  to 
fragments  (e.g.  “knowledge  type”  and  “material  use”)  and  other  parts  of  this 
ontology  can  be  used  for  the  development  of  instructional  bags  (see  for  more 
information:  section  3.1.4).  The  instructional  ontology  is  designed  with  a  focus  on 
maintenance  training,  and  contains  a  specific  (maintenance)  part  and  a  generic 
(instructional)  part.  This  ontology  can  be  re-used  in  other  training  domains 
requiring  a  focus  on  (complex)  cognitive  skills  for  system  maintenance.  However, 
the  generic  part  of  the  instructional  ontology,  can  be  re-used  in  any  other  training 
domain. 

The  domain  ontology  gives  a  conceptual  description  of  a  domain,  such  as  the 
description  of  a  system  or  domain.  The  domain  ontology  can  consist  of  a  generic 
and  specific  part.  Within  IMAT  the  generic  part  of  the  domain  ontology  is  called 
the  system  ontology.  The  system  ontology  describes  systems  in  terms  of  concepts 
applicable  to  various  domains,  such  as  unit  and  component.  The  specific  part  of  the 
domain  ontology  deals  with  specific  concepts  used  in  a  certain  domain,  and  varies 
per  domain.  The  domain  ontology  is  the  only  ontology  that  is  (for  the  largest  part) 
not  generic  and  will  have  to  be  redeveloped  for  a  new  technical  domain 
implementation.  For  the  RNLAF  it  is  important  to  know  how  much  time  and  effort 
it  will  take  to  create  new  domain  ontologies  and  how  the  ontology  development 
tool  can  assist  this  process. 
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3.1. 1.2  Document  Analysis 

The  document  analysis  process  distinguishes  the  following  four  phases 
(Anjewierden,  Wielinga  &  Kabel,  2001): 

•  Source  Material  Interpretation: 

AIDAS  enables  the  automatic  interpretation  of  Portable  Document  Format 
(PDF)  and  Extensible  Mark-up  Language  (XML)  documents. 

•  Discovery  of  the  logical  document  structure: 

Through  incremental  analysis  the  layout  structure  (sections,  tables,  images,  etc.) 
of  a  document  is  analysed  to  discover  the  logical  structure. 

•  Indexing  and  fragmentation  of  the  logical  document  structure: 

Ontologies  are  used  in  this  stage  to  index  text  and  image  fragments  created  by 
the  document  analysis  process.  The  ontology  descriptions  are  represented  as  a 
set  of  concepts  in  a  CML  format. 

•  Storage  of  the  document  fragments  in  a  multimedia  database: 

Finally  fragments  (and  their  indexes)  are  stored  in  the  Jasmine®  multi-media 
database. 

The  next  figure  gives  a  graphical  overview  of  the  AIDAS  main  window. 


Figure  3. 1:  AIDAS  tool:  Main  window 

Although  it  could  be  argued  that  the  document  analysis  process  does  not 
necessarily  need  a  user-interface  (it  is  an  automatic  process),  user  evaluations  in 
the  IMAT  project  clearly  showed  that  an  interface  should  be  provided  enabling 
human  inspection  during  the  document  analysis  process.  Therefore  the  AIDAS  tool 
provides  presentation  views  of: 

•  HTML  output  for  the  conversion  of  PDF  files  to  HTML, 

•  Analysed  text,  and 
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•  Analysed  images. 

The  images  are  converted  to  SVG  format  to  enable  image  analysis.  Through  the 
use  of  various  colours  an  AIDAS  user  can  see  which  parts  of  the  document  (text 
and  images)  are  analysed  in  what  way.  Examples  and  details  on  this  matter  will  be 
given  in  section  2.2.3:  Working  with  AIDAS. 

Two  other  tools  (or  functionalities)  are  integrated  in  AIDAS.  These  tools  are  the 
Fragment  Search  Tool  and  the  Ontology  Development  Support  Tool.  The  latter  tool 
will  be  further  described  in  section  2.2.2:  Ontology  Development. 

The  fragment  search  tool  enables  search  and  inspection  of  text  fragments  prior  to 
storage  in  the  central  database.  It  is  not  possible  to  export  text  fragments  retrieved 
with  this  tool  to  other  tools  or  environments.  For  this  purpose,  the  Authoring 
Environments  Interface  Tool  is  designed  in  IMAT  (see  next  section). 

3.1.2  Authoring  Environments  Interface  Tool 

The  general  purpose  of  the  Authoring  Environments  Interface  Tool  is  to  enable  the 
delivery  of  fragments  stored  in  the  multimedia  database  to  an  authoring 
environment  of  choice. 

Within  the  IMAT  project  it  was  decided  to  leave  the  authoring  environment  to  the 
free  choice  of  user  organisations.  They  can  then  choose  the  authoring  tools  they 
prefer  to  use  in  the  development  and  delivery  process  of  training  varying  from 
Word  or  PowerPoint,  to  tools  such  as  Toolbook  and  Authorware^  or  a  Learning 
Content  Management  System  (LCMS).* 5  At  the  start  of  the  IMAT  project  it  was 
strongly  felt  that  organisations  should  use  the  authoring  tools  they  were  using 
already  in  their  organisation  or  that  they  would  prefer  to  use.  “Prescribing” 
authoring  tools  to  the  IMAT  user  organisations/partners  would  limit  their 
organisational  and  instructional  design  choices  and  would  probably  not  have  been 
accepted  by  the  user  partners  in  the  IMAT  project  due  to  the  need  (and  risks)  to 
migrate  to  a  new  and  unfamiliar  authoring  environment. 

In  the  next  sections  the  main  features  of  the  authoring  environments  interface  tool 
are  described  (section  3. 1.2.1),  fragment  retrieval  (section  3. 1.2. 2)  and  fragment 
mark-up  provided  by  the  indexing  support  tool  (3. 1.2. 3). 


^  Toolbook  and  Authorware  are  two  examples  of  common  CBT  or  WBT  authoring  tools. 

5  An  LCMS  combines  LMS  functionalities  with  an  authoring  environment  and  a  content 
management  system. 
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3.1.2. 1  Main  features 

The  Authoring  Environments  Interface  Tool  enables  the  delivery  of  fragments  to 

an  external  authoring  environment  in  two  ways.  They  are: 

•  Copy,  cut  and  paste  fragments  (extracted  from  the  database)  in  the  Windows 
clipboard  or  save  them  directly  to  files. 

•  Through  the  use  of  instructional  bags:  this  provides  a  user  with  a  more 
advanced  way  of  creating  training  material.  See  also  section  2.1.4:  Instructional 
Scenarios  Tool. 

The  next  figure  gives  an  overview  of  the  main  window  of  the  Authoring 

Environments  Interface  Tool.  Functionality  provided  by  this  window  includes: 

•  Connection  with  the  IMAT  database, 

•  Fragment  Management:  this  includes  fragment  content  management,  search, 
retrieval  and  display  of  fragments,  and 

•  Instructional  Bag  Functionalities:  this  includes  a  link  with  the  Instructional 
Scenarios  Wizard  in  the  Instructional  Scenarios  Tool,  and  also  to  import  an 
existing  instructional  bag,  or  create  one  by  hand  directly  in  this  window.  An 
instructional  bag  can  be  used  as  basic  structure  to  query  the  database  for 
fragments. 


Figure  3.2:  Authoring  Environments  Interface  Tool:  Main  Window  Overview 
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The  Authoring  Environment  Interface  Tool  Window  is  divided  in  two  parts: 

•  Via  the  [set  criteria]  button  a  user  can  search  and  retrieve  fragments  (for 
example:  text  or  picture  fragments),  which  are  presented  in  top  frame  of  the 
window,  a  user  can  also  (by  selecting  one  of  the  other  buttons)  create  fragment 
mark-up  and  annotations  (see  for  more  information  on  this  matter  the  next 
sections), 

•  In  the  largest  frame  of  the  window  a  user  can  import  an  instructional  bag  or 
create  one  by  hand  directly  in  this  frame.  By  means  of  a  direct  link  from  this 
window  it  is  also  possible  to  create  an  instructional  bag  guided  by  the 
Instructional  Scenarios  Wizard.  When  an  instructional  bag  is  created,  it  is 
possible  through  the  Authoring  Environments  Tool  to  query  the  database,  and 
insert  fragments  directly  into  the  instructional  bag.  An  instructional  bag  also 
enables  the  (temporary)  storage  of  multiple  fragments  during  a  fragment 
search. 

See  for  more  information  on  the  creation  of  instructional  bags  and  the  Instructional 
Scenario  Wizard  section  3.1.3:  the  Instructional  Scenarios  Tool. 

3. 1.2.2  F ragment  retrieval 

The  next  figure  gives  an  overview  of  one  of  the  fragment  retrieval  options  provided 
in  the  IMAT  tools.  This  window  displays  the  Fragment  Retrieval  Window 
connected  to  the  Authoring  Environments  Tool. 


Figure  3.3:  Fragment  Search  and  Retrieval 

It  is  possible  to  retrieve  fragments  stored  in  the  database  from  the  original  technical 
manuals  or  fragments  which  have  been  altered  and  annotated  for  instructional 
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purposes,  such  as  instructional  fragments  or  instructional  bags.  The  fragment 
retrieval  tool  enables  a  search  in  the  IMAT  database  through  various  options  such 
as  topic  search,  keyword  search,  or  the  search  of  fragments  concerning  a  certain 
knowledge  type.  The  predefined  concepts  in  the  various  search  options  are  based 
upon  the  developed  ontologies  in  IMAT.  In  the  grey  frame  at  the  lower  left  side  of 
the  fragment  retrieval  window  a  search  log  presents  the  number  of  fragments  found 
in  the  database  given  certain  search  criteria. 

3.1.2.3  Indexing  support  tool:  fragment  mark-up 

Fragment  mark-up  can  be  seen  as  a  first  step  to  connect  a  fragment,  from  an 
instructional  perspective,  to  a  suitable  concept,  such  as  topic,  keyword,  knowledge 
type  or  fragment  use.  The  indexing  support  tool  supports  fragment  mark-up,  and 
this  functionality  is  also  provided  in  the  Database  Facilities  Tool.  Fragment  mark¬ 
up  can  be  done  both  at  the  level  of  a  single  fragment  or  multiple  fragments  at  once. 
The  next  figure  gives  an  overview  of  the  Fragment  Mark-up  Window. 


Figure  3.4:  Fragment  Indexing  Tool 

The  Fragment  Mark-up  Window  displays  various  categories  assisting  a  user  with 
predefined  concepts  related  to  the  domain  (derived  from  the  domain  ontology),  and 
possible  instructional  use  of  a  fragment.  The  concepts,  such  as  knowledge  type,  are 
presented  when  a  user  clicks  on  a  [+]  button,  after  which  a  selection  can  be  made, 
and  are  derived  from  the  instructional  ontology.  By  the  use  and  selection  of 
predefined  concepts,  fragment  search  and  retrieval  is  enhanced.  Not  only  because 
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the  same  concepts  are  used  between  different  users,  also  because  typing  mistakes 
are  prevented. 

3.1.3  Organisational  Memory  Tool 

The  general  purpose  of  the  organisational  memory  tool  is  to  assist  users  in  the 
process  of  creating  annotations  for  fragments,  and  to  facilitate  organisational 
memory  and  learning  on  the  use  of  instructional  material  in  an  organisation.  In 
essence,  the  organisational  memory  tool  facilitates  knowledge  sharing  of  fragments 
between  different  users.  Annotations  are  an  aspect  of  knowledge  sharing  and  give 
users  of  a  fragment  the  possibility  to  add  own  comments,  ideas,  opinions,  expertise 
and  experience  to  information  available  in  the  IMAT  database.  A  user  of  a 
fragment  could  be  anyone  with  access  to  the  IMAT  database.  A  user  could  have  for 
instance  the  role  of  creator,  designer  or  developer  of  a  fragment,  instructor,  or 
database  manager.  Even  trainees  or  end-users  of  fragments  could  be  given  rights  to 
access  the  database  to  create  annotations. 

Annotations  can  provide  rich  sets  of  information  on  the  use  of  a  piece  of 
information,  from  various  perspectives:  through  users  with  different  roles,  during 
different  lifestages  of  a  fragment,  such  as  prior  to  instructional  use  (design  and 
development  phase),  during  instructional  use  (training  phase)  or  afterwards 
(evaluation  phase).  Annotations  can  be  very  useful  for  an  organisation  since  they 
can  provide  a  structured  format  to  share  knowledge  and  experience  on  the  use  of 
instructional  material  (in  the  case  of  IMAT)  and  create  organisational  memory  and 
knowledge,  even  among  parties  not  working  on  the  same  physical  location. 

One  can  think  of  making  annotations  for  the  purpose  of  version  management, 
tracing  down  decisions,  communication  with  others,  and  providing  additional 
information  on  a  fragment.  Other  reasons  for  using  annotations  could  be 
authorisation  and  release  management  (when  should  a  certain  piece  of  information 
be  published,  and  for  whom).  Annotations  can  also  serve  as  a  memory,  e.g.  as 
reminder  that  the  fragment  contains  a  mistake  and  should  be  updated  with  a  new 
release  of  the  technical  manuals. 

In  IMAT  it  is  possible  to  create  annotations  for: 

•  Fragments,  stored  in  the  database; 

•  Instructional  bags  as  a  whole; 

•  Didactic  actions  within  instructional  bags; 

•  Media  fragments  linked  to  the  didactic  actions  in  an  instructional  bag. 

The  next  figure  gives  an  overview  of  the  Annotations  Window  of  the 
Organisational  Memory  Tool. 
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Figure  3.5:  Organisational  Memory  Tool:  The  Annotations  Window 

In  the  I  MAT  annotations  system  four  categories  are  distinguished: 

•  Usability:  this  category  contains  items  to  describe  a  fragment,  such  as  reading 
efficiency,  understanding  efficiency,  misunderstanding  and  recovery  strategies, 
subjective  satisfaction  and  target  group. 

•  Usefulness:  this  category  contains  items  to  describe  a  fragment,  such  as 
relevance,  internal  and  external  consistency,  tractability  and  course 
appropriateness. 

•  Evaluation  context:  this  category  is  important  to  fill  in  as  a  reminder  for  the 
creator  of  an  annotation  and  others  reading  the  annotation,  because  it  provides 
information  such  as  who  made  the  annotation,  when  and  why. 

•  Additional  Information:  this  category  provides  the  user  (amongst  others.)  with 
a  free  text  option  to  fill  in  any  other  information  on  a  fragment  that  might  be 
relevant. 

Each  of  these  four  categories  is  presented  on  a  tab  page  in  the  Annotation  Window, 
and  drop  down  boxes  with  pre-set  options  or  free  text  options  are  provided  to 
annotate  a  fragment. 

3.1.4  Instructional  Scenarios  Tool 

The  Instructional  Scenarios  Tool  supports  a  user  with  the  creation  of  a  standard 
structure  for  a  specific  lesson  (skeletal  lesson).  In  the  Instructional  Scenarios  Tool 
a  user  is  guided  and  assisted  by  an  Instructional  Scenarios  Wizard. 
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The  assistance  provided  by  the  instructional  scenarios  wizard  to  a  user  can  be 
compared  to  the  way  a  filter  works.  After  the  selection  of  a  learning  goal,  the  first 
step  in  the  creation  of  an  instructional  bag,  the  wizard  filters  options  based  upon 
the  first  selection  and  prior  to  the  next  step  in  the  process  presents  only  those 
options  making  sense.  This  process  is  repeated  until  all  the  steps  necessary  for  a 
complete  instructional  bag  are  filled  in.  The  wizard  only  guides  the  process  by 
eliminating  false  or  contradictory  options  but  will  not  interfere  with  choices  made 
by  an  instructional  designer.  However,  the  set-up  is  that  of  a  wizard,  meaning  that 
it  guides  the  steps  of  a  user  in  a  directive  manner  and  prevents  a  user  from  making 
mistakes  as  much  as  possible. 


The  next  figure  gives  an  example  of  one  of  the  steps  in  the  wizard,  where  a  user 
can  select  a  predefined  concept.  The  figure  gives  an  overview  of  the  first  step  of 
the  instructional  scenario  wizard:  the  learning  goal  choice. 


Figure  3.6:  Learning  Goal  Choice 

The  concepts  presented  in  the  various  steps  of  the  instructional  scenarios  wizard 
are  derived  from  the  instructional  ontology.  The  next  figure  gives  an  overview  of 
the  instructional  scenarios  wizard  with  a  filled  in  scenario  for  an  instructional  bag. 
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Figure  3.7:  Instructional  Scenarios  Tool:  Overview  of  a  filled  in  scenario  constructed 
with  the  help  of  the  Instructional  Scenarios  Wizard. 


The  next  steps  are  defined  and  guided  by  the  instructional  wizard  in  the  process  of 

creating  an  instructional  bag  (Barnard,  Desmoulins  &  Grandbastien,  2002).  They 

are: 

•  Set  a  learning  goal:  As  first  step,  a  user  can  select  a  learning  goal  from  a  list  of 
learning  goals  designed  for  technical  training. 

•  Fragment  Description:  The  Wizard  will  then  automatically  select  a  fitting 
fragment  description. 

•  Determine  the  knowledge  type:  As  the  next  step,  a  user  has  to  select  the 
knowledge  type  a  student  needs  to  obtain,  such  as  “performing  skills”  or 
“procedural  knowledge”.  A  selection  can  be  made  from  a  list  of  knowledge 
type  concepts  derived  from  the  instructional  ontology. 

•  Choose  an  instructional  strategy:  The  next  user  step  is  to  select  an  instructional 
strategy  that  is  suited  for  the  intended  lesson,  students  and  classroom 
organisation.  Again,  selecting  one  of  the  options  presented  by  the  wizard  has  to 
be  done.  Note  that  the  wizard  acts  as  a  filter  and  will  present  only  those 
instructional  strategies  relevant  and  not  contradicting  previous  choices. 

•  Determine  a  set  of  instructional  activities:  the  wizard  will  then  automatically 
select  a  set  of  instructional  activities.  The  concepts  presented  are  derived  from 
the  user’s  choice  for  a  specific  instructional  strategy. 

•  Instructor  and  learner  actions:  The  wizard  will  also  present  a  sequential  list  of 
instructor  and  learner  actions  that  are  connected  to  the  instructional  activities. 

•  Create  an  action  description:  A  user  can  (optionally)  describe  an  action  further. 
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•  Material  Use:  This  is  finally  presented  by  the  Wizard  as  a  further  refinement  of 
instructor  and  learner  actions,  and  describes  the  type  of  material  to  be  used  in  a 
certain  lesson. 

It  is  possible  to  use  the  Instructional  Scenarios  Tool  as  a  stand-alone  tool  or  in 
conjunction  with  the  IM  AT  database.  The  latter  is  possible  through  the  export  of 
instructional  bags  to  the  Authoring  Environment  Tool,  where  an  instructional  bag 
can  be  used  to  query  the  database  for  fragments. 

3.1.5  IMAT  Data-handling 

This  section  will  describe  aspects  related  to  the  IMAT  Database  and  the  adjacent 
Database  Facilities  Tool  to  give  a  complete  description  of  the  IMAT  concept,  since 
they  both  fulfil  a  central  role  in  using  the  IMAT  tools. 

3.1.5. 1  Jasmine®:  the  multimedia  database 

In  the  IMAT  project  several  commercial  off  the  shelf  software  products  were  used, 
among  which,  the  Jasmine®  database.  This  is  an  object-oriented  relational  database 
used  to  store  fragments  in  various  stages  of  their  lifecycle.  Jasmine®  was,  amongst 
other  factors,  chosen  for  its  ability  to  handle  multimedia  formats.  The  IMAT 
database  is  the  central  unit  where  fragments  are  stored,  retrieved  and  managed  in 
their  various  states  during  the  IMAT  process  of  creating  fragments,  instructional 
fragments  and  instructional  bags. 

3.1.5.2  The  Database  Facilities  Tool 

To  be  able  to  manage  the  fragments  in  various  stages  and  forms  in  the  database,  i.e. 
version  management  and  perform  other  management  tasks,  such  as  loading  or 
clearing  a  context  and  editing  domain  ontologies  the  Database  Facilities  Tool  has 
been  developed. 

The  Database  Facilities  Tool  supports: 

•  Ontology  modification:  the  database  facilities  tool  enables  the  modification  of 
an  existing  ontology  stored  in  the  database. 

•  Fragment  mark-up.  It  is  possible  to  create  fragment  mark-up  through  the  same 
user  interface  as  provided  in  the  Authoring  Environment  Tool,  see  figure  3.3. 

•  Textual  Fragment  Upgrade:  It  is  also  possible  to  perform  a  textual  upgrade  on  a 
text  fragment  in  any  state  of  the  lifecycle  of  the  fragment  (draft,  original  or 
instructional  material)  to  correct  mistakes  in  a  fragment.  Since  typing  errors  are 
one  of  the  more  common  mistakes  in  technical  manuals,  this  option  can  be 
very  useful,  especially  in  domains  where  the  update  cycle  of  technical  manuals 
takes  considerable  time. 

•  Version  management:  The  database  facilities  tool  enables  version  management 
of  fragments  derived  from  the  technical  manual  and  “edited”  instructional 
fragments. 
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•  Annotations:  in  the  database  facilities  tool  annotations  can  be  created,  for: 

•  Fragments,  derived  from  the  technical  manual,  and 

•  Updated,  changed  or  newly  created  fragments.  The  user  interface  provided 
for  the  creation  of  annotations  in  the  Database  Facilities  Tool,  is  the  same 
as  in  the  Authoring  Environment  Tool,  see  figure  4. 

As  described  in  chapter  2  on  the  lessons  learned  in  the  European  IMAT  project  and 
GGW  De  Peel  an  important  remaining  problem  concerns  the  automatic  support  of 
changes  and  updates  in  technical  domains  by  IMAT.  However,  in  domains  without 
large  amounts  of  updates  per  “update  moment”  it  is  possible  (and  manageable) 
with  the  version  management  facilities  provided  by  the  database  facilities  tool,  to 
search  and  retrieve  instructional  fragments  and  update  them  manually  if  necessary. 
This  type  of  change  and  update  management  will  prevent  the  (manually  added) 
instructional  indexes  from  getting  lost,  as  would  be  the  case  after  an  automatic 
rebuild  or  refill  of  the  Jasmine®  database. 


3.2  Tools  Analysis 

The  next  sections  describe  the  results  from  the  study  concerning  a  new  domain 
implementation  and  focuses  on  the  usability  of  the  current  version  of  IMAT  tools. 

3.2.1  Purpose  of  a  new  domain  implementation 

Considering  that  the  IMAT  tools  were  designed  and  developed  during  the 
European  IMAT  project,  it  is  important  to  understand  that  the  goal  of  this  project 
was  not  to  construct  a  set  of  fully  developed  commercial  tools,  but  rather  a  set  of 
prototypes  sufficiently  developed  and  evaluated  to  start  the  construction  of 
commercially  exploitable  tools.  However,  as  part  of  the  IMAT  follow-up  research 
it  was  decided  to  study  whether  the  Pfinal  set  of  tools  would  or  would  not  be 
directly  usable. 

During  the  first  part  of  the  European  IMAT  project  the  Flycatcher  domain  was 
used  as  the  example  domain  for  testing  the  tools.  When  it  became  clear  that  the 
RNLAF  would  cease  to  provide  training  for  Flycatcher  maintenance  personnel,  this 
domain  was  abandoned  and  the  HAWK  domain  was  implemented.  Many  technical 
problems  occurred  during  the  implementation  of  the  Hawk  domain  in  the  IMAT 
prototype  tools  due  to  a  combination  of: 

•  technical  problems  and  lacking  stability  of  the  prototype  tools  that  were  still 
under  development; 

•  no  emphasis  on  user  friendliness  and  exhaustive  manuals  for  these  tools  under 
development; 

•  quality  of  the  source  material:  the  electronic  HAWK  manuals  were  scanned 
from  the  paper  version;  problems  arose  because  the  text  was  not  clear  (letters 


TNO  report 


30 


TNO-DV1  2004  A169 


incomplete  or  too  close  together),  irregular  position  of  pictures  (skewed  on  the 
page),  etc. 

It  was  not  clear  which  of  these  factors  had  caused  the  problems  and  in  how  far  the 
problems  had  been  solved  in  the  final  prototypes  that  were  delivered  after  the  last 
evaluation  rounds.  Therefore,  we  decided,  as  part  of  the  IMAT  follow-up  project, 
to  try  to  implement  a  new  domain.  More  specifically,  we  wanted  to  investigate  the 
effort  required  in  adding  a  new  domain  to  the  IMAT  tools.  We  aimed  to  gain 
insight  in  the  usability  of  the  current  version  of  the  tools,  and  how  feasible  it  is  for 
non-IMAT  experts  to  add  a  new  domain  to  the  tools.  We  had  the  following 
questions  in  mind: 

•  How  difficult  is  it  to  create  a  (new)  domain  ontology? 

•  How  difficult  is  it  to  implement  the  domain  ontology  in  the  tools? 

•  How  difficult  is  it  to  tune  the  AIDAS  tools  to  new  source  material? 

•  How  difficult  is  it  to  get  the  text  and  image  analysis  working  in  a  new  domain? 

•  How  stable  are  the  analyses,  do  they  render  the  same  results  over  several  trials? 

on  different  computers? 

•  Do  the  analyses  work  on  all  parts  of  the  text  and  images? 

•  Do  the  fragments  get  into  the  database? 

•  Can  we  easily  retrieve  the  fragments  again  from  the  database? 

3.2.2  Method 

For  the  purpose  of  this  study,  a  new  domain  has  been  added  to  the  IMAT  tools.  It 
was  anticipated  that  help  might  be  needed  from  other  IMAT  partners6  that  had 
been  responsible  for  the  development  of  the  tools.  In  that  case,  a  clearer  view  on 
the  amount  of  help  and  related  costs  for  a  new  domain  implementation  could  be 
estimated. 

The  findings  and  experiences  with  regard  to  the  research  questions  are  described 
according  to  the  steps  involved: 

•  Selecting  a  domain  (see  section  3.2.3); 

•  Developing  the  domain  ontology  (see  section  3.2.4) 

•  Adding  new  domain  files  to  the  IMAT  directory  (see  section  3.2.5) 

•  Starting  the  text  and  image  analysis  (see  section  3.2.6) 

•  Retrieving  fragments  (see  section  3.2.7) 

•  Exporting  fragments  (see  section  3.2.8) 

3.2.3  Selecting  a  new  domain 

Since  the  questions  were  mainly  of  technical  nature  and  did  not  address  specific 
RNLAF  issues,  it  was  decided  to  try  implementing  a  new  domain  without  the 
necessity  of  using  RNLAF  domain  expertise.  For  this  reason,  and  to  make  the 


6 


Especially,  ISLA  at  the  University  of  Amsterdam. 
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results  comprehensible  for  non-domain  experts,  we  searched  for  a  technical 
maintenance  domain  that  was  limited  in  complexity.  This  initiated  a  search  for 
source  material,  an  electronic  technical  manual,  compliant  with  the  following 
demands: 

•  Non-confidential; 

•  Available  for  free; 

•  PDF  or  XML  format; 

•  Containing  both  text  and  pictures; 

•  Good  quality; 

•  Realistic  size,  but  not  too  big. 

A  search  with  Google  on  the  Internet,  according  to  these  criteria  resulted  in  the 
selection  of  a  technical  manual,  in  PDF  format.  The  manual  was  publicly  available 
and  downloaded  from  www.goodvearaviation.com/tirecare.html.  The  selected 
document,  ‘Aircraft  Tire  Care  and  Maintenance’  concerns  the  aircraft  tire 
maintenance  domain.  It  is  a  technical  maintenance  domain  that  is  not  too  complex 
(non-experts  will  fairly  easily  become  familiar  with  the  domain). 

What  is  interesting  to  note,  is  the  fact  that  this  manual  is  written  in  a  task-oriented 
manner,  which  is  very  different  from  the  more  system-oriented  manuals  used  in  the 
European  IM  AT  project.  For  this  reason,  the  following  additional  questions  came 
to  mind: 

•  Do  the  generic  ontologies  (developed  during  the  European  IMAT  project) 
“match”  sufficiently  with  this  new  domain? 

•  Should  the  (to  be  developed)  domain  ontology  contain  a  system  and  a  task- 
oriented  part? 

•  Could  the  (generic)  instructional  ontology  be  used  to  index  the  task-oriented 
parts  of  the  technical  manual? 

3.2.4  Developing  the  domain  ontology 

The  second  step  was  to  create  the  domain  ontology  for  the  selected  domain.  The 
ontology  should  describe  the  domain  in  a  systematic  manner  in  order  to  support  the 
indexing  of  the  technical  manual  into  meaningful  fragments  and  to  index  them  on 
topic.  The  system  ontology  describes,  in  essence,  systematically  the  compounds  of 
the  system. 
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The  development  of  a  domain  specific  ontology  is  assisted  by  the  IM  AT  tools  in 
the  following  three  ways: 

1.  A  new  ontology  can  be  created  automatically  from  a  structured  part  lists  of 
the  equipment,  e.g.  from  a  table  of  contents  from  technical  manuals, 
followed  by  manual  fine-tuning. 

2.  A  new  ontology  can  be  created  manually  from  scratch  using  the  ontology 
development  support  tool 

3.  Existing  ontologies,  stored  in  the  database,  can  be  modified  using  the 
ontology  development  support  tool  or  the  database  facilities  tool. 

In  the  study,  we  reused  an  existing  ontology  to  create  a  new  one  for  the  selected 
PDF  manual  on  aircraft  tires  maintenance  the  aircraft  tire  domain.  We  followed  the 
following  steps  in  creating  the  new  domain  ontology: 

First,  the  IMAT  generic  system  ontology  was  analysed.  This  has  been  done  by 
analysing  the  specification  of  the  generic  ontology  in  the  CML  file  and  creating  a 
paper-based  graphical  representation  of  the  aircraft  tires  domain  to  visualise  the 
relevant  ‘is-a’  and  ‘part-of’  relations  between  system  compounds,  subsystems,  and 
system  locations.  This  representation  provided  a  systematical  framework  of  all 
kind  of  relations  that  a  system  ontology  could  contain. 

Next,  the  structure  of  the  specific  domain  ontology  developed  in  the  IMAT  project 
for  the  Hawk  domain  was  analysed  to  see  how  compounds  of  the  Hawk  system 
were  linked  to  the  compounds  of  the  generic  ontology.  This  provided  a  useful 
example  of  a  domain  ontology  and  how  it  is  mapped  on  the  generic  system 
ontology,  the  compounds  and  several  identity  relations.  The  graphical 
representations  of  the  generic  ontology  and  the  Hawk  ontology  served  as  reference 
for  building  the  new  domain  ontology  and  implementing  the  relations. 

Then,  a  paper-based  graphical  representation  of  the  new  domain  ontology  (‘the 
aircraft  tire  ontology’)  was  created  according  to  systematic  descriptions  and 
graphical  representations  of  the  system  in  the  manual.  Often,  the  domain  ontology 
can  be  generated  on  basis  of  an  index,  a  table  of  contents,  or  a  tree  diagram  of  a 
system  overview  in  the  manual.  These  general  indexes  serve  as  good 
representations  of  a  compound  system  or  at  least  as  good  starting  point.  However, 
in  the  aircraft  tire  maintenance  domain  this  was  only  partly  true.  The  manual  is 
written  from  a  task-oriented  perspective  and  the  table  of  contents  reflects  the 
maintenance  tasks  involved  rather  than  units  that  the  system  is  compounded  of.  For 
example,  one  chapter  describes  the  inspection,  repair  &  storage  tasks  and  another 
chapter  the  preventive  maintenance  tasks  for  various  system  parts. 

Considering  that  the  generic  IMAT  domain  ontology  describes  the  system 
compounds  rather  then  the  maintenance  tasks  involved,  the  domain  ontology  could 
not  be  generated  on  basis  of  the  table  of  contents.  In  the  end,  other  parts  of  the 
document,  like  lists  of  compounds  and  graphical  overviews  of  system  parts  (such 
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as  depicted  in  the  figure  below)  served  as  a  source  of  input  for  creating  the 
ontology. 
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Figure  3.8 :  Aircraft  tire  parts  overview  from  the  technical  manual 

A  problem  came  up  when  we  wanted  to  implement  some  location  relations:  For 
example,  the  aircraft  tire  manual  describes  areas  and  (sub)systems  that  are  located 
in  such  an  area  (e.g.  bead  area).  However,  the  Hawk  ontology  (which  served  as  an 
example  of  how  to  implement  the  relations)  does  not  contain  any  location  relations. 
Additionally,  the  IMAT  manuals  also  do  not  address  these  more  complex  relations. 
In  the  end,  we  decided  not  to  implement  the  location  relations  in  the  final  ontology. 

The  resulting  graphical  presentation  of  the  ontology  with  the  relevant  ‘is-a’  and 
‘part-of  relations  has  been  enhanced  with  synonyms  and  abbreviations  where 
relevant.  The  figure  below  shows  a  part  of  the  ontology  in  an  unfinished  state  (the 
blocked  boxes  represent  the  generic  system  ontology). 
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Figure  3.9:  Graphical  representation  of  a  part  of  the  new  domain  ontology 

The  graphical  representation  of  the  ontology  served  as  reference  material  when 
adding  the  ontology  to  the  IM  AT  tools  and  discussions  later  on. 

Finally,  in  order  to  load  the  ontology  in  the  AIDAS  tool,  the  domain  ontology 
needed  to  be  converted  into  a  CML-file,  meaning  that  the  set  of  domain  concepts 
(as  represented  in  the  graphical  representation)  needed  to  be  represented  in  the 
Conceptual  Modelling  Language.  We  tried  to  use  the  ontology  development  tool  in 
AIDAS  to  create  the  new  ontology.  This  ontology  development  support  tool  that  is 
integrated  in  the  IM  AT  tools  was  developed  to  assist  the  creation  and  modification 
of  domain  specific  ontologies.  The  figure  below  gives  an  overview  of  the  Ontology 
Development  Support  Tool,  depicting  parts  of  the  new  domain  ontology  for  aircraft 
tire  maintenance. 
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Figure  3. 10:  Ontology  Development  Support  Tool:  showing  parts  of  the  new  domain 
ontology 

It  however  appeared  that  the  editor  suffers  from  usability  problems.  The  main 
problem  was  that  the  editor  does  not  provide  a  complete  overview  of  the  relations 
between  system  parts.  The  window  displays  four  separate  browsers;  an  is-a  concept 
browser,  a  part  of  concept  browser,  a  words  browser,  and  a  next  words  browser.  As 
a  consequence,  contextual  information  is  lacking. 

In  the  end,  we  decided  to  create  the  Conceptual  Modelling  Language  manually  by 
customising  the  existing  CML  file  that  was  created  for  the  HAWK  domain. 
Although  it  proved  to  be  easy  to  slip  up,  it  appeared  not  to  be  so  difficult.  We 
learned  that  it  requires  much  precision  in  placing  the  quotes  in  the  programming 
code.  The  figure  below  shows  a  part  of  the  programming  code  in  the  CML 
language. 
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1  File  Edit  Format  View  Help 

- 

/*  cml  model  * 

Domain:  "multiple" 

Generated  by  CPL2CML  Generator 
created:  9-1-2001  (Bob  wi el  Inga) 


KNOWLEDGE-MODEL  ‘generic  systems'; 

DOMAIN-KNOWLEDGE  'generic  system'; 

ONTOLOGY  root; 
concept  entity; 

END  CONCEPT 
END  ONTOLOGY 


ontology  ' system  types ' ; 
definitions:! 

concept  'system  type’; 
sub-type-of:  entity; 
end  concept  'system  type'; 

concept  'compound  system'; 

description:  "a  complex  functional  object  consisting  of  one  or  more  pieces  of  equipment  each 
functions.  The  pelces  of  equipment  do  not  have  to  be  located  in  one  place,  more  likely,  they 
number  of  different  locations.  The  KLU  SHORAD  is  an  example  of  a  compound  system,  so  is  the 
Control  System"; 
sub-type-of:  ' system  type' ; 

has_parts :  equipment; 

END  concept  'compound  system'; 

a|! 


Figure  3.11 :  Part  of  the  CML  programming  code 

Conclusions  on  the  development  of  a  new  domain  ontology: 

•  The  creation  of  a  new  ontology  was  not  very  difficult.  It  took  us  about  32  man¬ 
hours  to  create  a  paper-based  version  for  an  electronic  aircraft  maintenance 
manual  of  49  pages.  In  reality,  the  modelling  of  the  domain  ontology  would  be 
a  job  for  a  domain  expert  who,  because  of  his/her  extensive  domain  knowledge 
would  have  little  difficulty  in  modeling  the  domain. 

•  The  conversion  of  the  paper-based  ontology  in  CML  was  not  very  difficult.  It 
took  us  about  8  man-hours.  However,  the  implementation  of  the  graphical 
representation  of  the  ontology  requires  some  knowledge  of  software 
programming,  which  will  not  always  be  available  to  the  domain  experts. 

•  The  ontology  development  tool  in  AIDIAS  was  too  difficult  to  use.  It  lacks  a 
coherent  overview  of  the  hierarchical  relations.  However,  implementing  the 
domain  ontology  directly  in  a  copy  of  another  domain  ontology  file  was  not 
difficult  for  someone  with  programming  experience.  One  has  to  think  of 
naming  conventions  and  quoting. 

•  There  was  not  enough  support  on  how  to  implement  more  complex  relations: 
Firstly,  the  example  ontology  of  the  HAWK  domain  that  was  available  did  not 
define  any  complex  relations  (e.g.  location  relations).  And  secondly,  there  was 
no  manual  available  that  provided  guidance  on  this  issue. 

•  Also,  there  was  not  enough  support  for  how  to  deal  with  task-oriented  manuals, 
like  the  aircraft  tire  manual.  Where  the  table  of  contents  of  system  oriented 
manuals  reflects  a  hierarchical  decomposition  of  the  system,  the  table  of 
contents  of  the  aircraft  tire  manual  reflects  a  decomposition  of  maintenance 
tasks.  Although  the  latter  situation  is  more  ideal  from  an  instruction 
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perspective,  it  is  harder  to  create  an  ontology  on  the  basis  of  such  a  manual 
(note  that  in  an  ideal  situation  the  expert  uses  the  table  of  contents  of  the 
manual  as  the  basis  for  creating  the  system  ontology). 

3.2.5  Adding  new  domain  files  to  IMAT  directory 

The  next  step  was  to  upload  the  CML  file  together  with  the  aircraft  tire  manual  in 
the  AIDAS  tool.  This  section  describes  in  detail  how  we  added  the  new  domain  to 
the  IMAT  directory.  Note  that  the  drive  and  directory  names  are  specific  for  the 
software  configuration  that  we  used,  and  will  be  different  on  other  PCs. 

3.2.5.1  Create  a  new  domain  directory 

On  the  D-drive  of  the  computer  was  a  directory  called  IMAT  (D/IMAT).  The 
folder  contained  two  subdirectories,  ‘applications’  (D/IMAT/applications)  and 
‘Pfinaf  (D/IMAT/Pfinal).  The  applications  folder  contained  domain  specific  files 
(like  ontologies,  source  documents,  DAS  files)  for  several  domains.  The  Pfinal 
folder  contained  the  program  files  to  run  the  IMAT  tools.  We  created  new 
subdirectories  in  the  applications  folder,  which  were  called:  ‘das’, 
‘document_status’,  ‘images’,  ‘indexes’,  ‘odlql’,  ‘ontology’,  ‘source_documents’, 
‘text_fragments’,  ‘transformed_documents’,  and  ‘xml’.  We  found  out  that  an  easy 
way  to  create  the  new  domain  directory  was  copying  one  of  another  domain 
directory,  pasting  it  in  the  applications  directory  and  renaming  it.  After  having 
given  it  the  new  domain  name,  the  ‘read  only’  attribute  was  turned  off  in  the  folder 
settings  (available  via  the  right  mouse  menu:  select  ‘properties’  and  deselect  the 
‘read-only’  attribute  from  the  general  tab). 

3.2.5.2  Add  a  new  source  document 

For  adding  the  new  source  document  (the  aircraft  tire  manual)  to  the  AIDAS  tool, 
we  opened  the  directory  D/IMAT/applications/source_documents,  deleted  all 
existing  source  documents  in  the  folder  and  pasted  the  new  source  document  (the 
PDF-file)  in  the  corresponding  subdirectory.  Note  that  the  tool  does  load  PDF  and 
XML  files  (depending  on  type),  but  no  Word  files. 

Add  CML  file 

In  order  to  add  the  new  ontology  to  the  AIDAS  tool  we  opened  the  directory 
D/IMAT/applications/ontology,  deleted  the  existing  files  in  that  folder  and  added 
the  new  CML  file  to  directory. 

Fine  tune  DAS  file 

Finally,  the  DAS  file  needed  to  be  fine-tuned.  The  DAS  file  was  located  in  the 
directory  D/IMAT/applications/das.  The  main  task  was  to  make  the  DAS  file  call 
(activate)  the  new  domain  ontology.  For  this  reason,  we  customised  the  program 
code  in  an  existing  DAS  file:  We  opened  the  DAS-file  and  replaced  the  name  of 
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the  CML-file  in  the  top-line  of  the  code  (see  figure  below:  Replace 
‘vliegtuigbanden.cml’  by  the  name  of  your  new  CML-file). 


Figure  3.12:  Program  code  in  the  DAS  file 


3.2.6  Uploading  the  ontology  to  start  text  and  image  analyses 

The  next  step  was  to  start  the  text  analysis  of  the  new  manual  using  the  new 
ontology.  To  load  the  new  ontology  in  the  AIDAS  tool,  we  changed  the  ‘context’ 
setting  into  the  new  domain  context;  ‘banden’.  When  having  set  the  context,  the 
ontology  loads  automatically. 

By  looking  in  the  SWI-Prolog  screen,  we  were  able  to  check  whether  the  ontology 
indeed  was  loaded.  The  Prolog  log  file  however  does  not  provide  feedback  about 
all  relevant  actions,  e.g.  the  Log  file  does  not  always  show  that  the  CML/Das  file  is 
loaded.  The  next  figure  shows  the  AIDAS  menu  for  loading  the  ontology  plus  the 
SWI-Prolog  code. 
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Copyright  1993-2001  University  of  Amsterdam 
Copying  GPL-2  (see  file  COPYING  or  wvv.gnu  org) 

The  host-language  is  SWI -Pro log  version  4  0  11 
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1  7- 
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Figure  3. 13:  AIDAS  menu  for  loading  a  domain  ontology 


Next,  we  loaded  the  PDF  manual  by  selecting  the  set  document  button,  selected  the 
document  and  the  page  numbers  that  we  wanted  to  be  analysed.  In  order  to  save 
time  we  selected  a  limited  number  of  pages.  During  the  analysis  process,  we  were 
confronted  with  some  faults  in  the  programming  code  of  the  CML  file.  All  faults 
that  were  found  concerned  incorrectly  placed  quotes.  With  some  help  of  one  of  the 
implementers  of  the  tools  (from  ISLA  at  the  University  of  Amsterdam)  we  were 
able  to  correct  them. 

Due  to  the  fact  that  AIDAS  does  not  provide  meaningful  error  messages,  it  was 
hard  to  track  down  simple  faults  in  the  CML-file,  like  incorrect  quotes,  typing 
faults,  and  spaces  in  titles.  For  this  reason  it  appeared  better  to  experiment  with  a 
restricted  number  of  pages  first  (by  setting  the  page  range),  until  all  errors  were 
fixed. 

The  text  and  picture  analysis  was  not  successful.  The  picture  analysis  did  not  work 
and  the  text  analysis  did  not  produce  correct  results.  The  figure  below  shows  a 
result  of  the  analysis.  It  appeared  that  the  tool  is  very  sensitive  for  the  quality  of  the 
PDF-file. 
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Figure  3. 14:  Results  of  the  text  and  picture  analyses  in  AIDAS 

Since  the  fragmentation  process  did  not  succeed,  we  were  not  able  to  store  any 

indexed  fragments  in  the  Jasmin  database. 

The  conclusions  of  the  document  analysis  process  are: 

•  The  interface  of  the  AIDAS  tool  is  not  very  easy  to  use.  It  is  not  obvious  how  a 
new  domain  ontology  can  be  loaded,  nor  meaningful  error  messages  and 
feedback  on  user  actions  is  given.  Additionally,  the  IM AT  manuals  do  not 
provide  help  on  this.  For  this  reason,  it  takes  a  lot  of  effort  to  find  out  what 
actually  is  the  problem.  Only  users  with  programming  experience  will  benefit 
from  some  feedback  provided  in  the  SWI-Prolog  screen.  In  order  to  save  time, 
while  testing  a  new  CML-file,  it  is  recommended  to  start  the  analyses  with  a 
restricted  number  of  pages; 

•  So  far,  the  document  analysis  has  not  produced  correct  results  (fragments)  from 
the  structure  and  text  analysis.  Given  the  type  of  problems,  the  expectation  is 
that  with  some  help  of  ISLA  further  fine-tuning  of  the  DAS  file  will  be 
necessary  to  obtain  proper  results  of  the  structure  and  text  analysis.  This  should 
be  done  by  AIDAS  experts  and  given  the  type  of  problems  as  well  as  the  well 
defined  and  structured  format  of  the  air  tire  manual  this  did  not  appear  to  be 
problematic  at  the  time  of  writing  of  this  report; 

•  The  picture  analysis  did  not  work,  probably  due  to  the  extensive  grey  scales  of 
the  pictures  in  the  air  tire  manual  from  Goodyear  (most  common  error  reference 
type  in  AIDAS  was  “picture  is  too  grey”)-  Problems  with  the  image  analysis 
have  not  been  resolved  during  the  writing  of  this  report  and  at  this  point  it  is  not 
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possible  to  give  a  proper  estimation  on  the  exact  scale  (major  or  minor)  of  these 
problems.7 

3.2.7  Retrieving  fragments 

The  next  step  in  the  new  domain  implementation  process  would  be  to  retrieve  the 
stored  fragments  from  database  and  to  see  whether  instructional  authors  would  be 
able  to  find  and  use  the  fragments  about  the  desired  topics.  However,  given  the  fact 
that  the  fragmentation  failed  and  no  new  domain  fragments  were  stored  in  the 
Jasmine  database,  this  part  of  the  study  could  not  be  performed  with  the  new 
domain.  The  ideas  described  below  are  based  on  findings  during  the  tool’s 
evaluation  study  plus  the  results  of  the  evaluations  performed  at  GGW  De  Peel 
during  the  European  IMAT  project  (Verstegen,  Veldhuis,  Staalstra  &  Hendriks, 
2001). 

Indexing  of  fragments 

The  AIDAS  tool  stores  the  fragments  according  to  topic,  based  on  the  structure  of 
the  document  and  the  topics  in  the  domain  ontology.  As  described  earlier,  the 
domain  ontology  solely  describes  the  system  compounds  and  the  relations  between 
them.  This  normally  corresponds  with  the  structure  of  the  electronic  technical 
manuals  for  system  maintenance.  However,  the  aircraft  tire  maintenance  manual  is 
written  from  a  task-perspective  and  is  structured  according  to  the  tasks  involved  for 
several  system  parts.  Because  the  generic  IMAT  domain  ontology  is  limited  to 
representing  structural  relations  between  system  compounds,  the  relations  between 
the  maintenance  tasks  and  the  system  parts  (as  described  in  the  manual)  could  not 
be  incorporated  in  the  domain  ontology.  However,  to  support  searching  in  a 
maintenance  context,  the  fragments  may  also  need  to  be  enhanced  with  metadata 
on  how  the  fragments  are  used  in  a  maintenance  context.  During  the  study,  the 
question  came  up  whether  indexing  of  fragments  according  to  system  compounds 
would  be  sufficient  for  the  aircraft  tire  maintenance  manual.  We  suspect  that  the 
task-oriented  descriptions  in  the  manual  can  be  very  relevant  for  searching  in  a 
maintenance  context.  In  the  future,  it  may  be  interesting  to  consider  an  additional 
task-part  added  to  the  system  ontology. 

Adding  mark-up 

It  appeared  desirable  to  add  mark-up  on  instructional  intentions.  The  instructional 
ontology  can  be  used  for  this  purpose.  It  allows  indexing  of  fragments  with  a  role- 
based  mark-up,  about  the  way  the  fragments  are  going  to  be  used  in  instruction, 
e.g.  ‘is  a  fragment  on  tread  wears’,  ‘can  be  used  as  an  example  inspection  task’. 
Also  here,  we  suspect  that,  the  indexing  of  fragments  according  to  maintenance 
task  involved  (as  described  above)  would  support  the  users  in  the  process  of 
retrieving  the  right  fragments  and  to  add  the  mark-up. 


7 


ISLA  and  the  University  of  Amsterdam  continued  to  work  on  picture  analysis 
(outside  the  scope  of  the  IMAT  projects)  and  have  developed  promising  solutions 
to  enhance  the  quality  of  picture  analysis. 
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Using  instructional  bags 

More  likely  however  is,  given  the  results  of  the  GGW  de  Peel  study,  that 
instructors  will  use  instructional  mark-up  to  enhance  larger  chunks  of  learning 
material  such  as  the  instructional  bags. 

3.2.8  Exporting  fragments 

Exporting  fragments  (manually)  is  supported  in  various  ways  in  the  IMAT  tools 
and  appears  to  be  very  easy.  The  results  of  the  user  evaluations  of  the  GGW  de 
Peel  and  other  participating  countries  in  the  European  IMAT  Project  also  confirm 
this  (See  Chapter  2). 

3.2.9  Conclusions 

How  difficult  is  it  to  create  a  (new)  domain  ontology? 

The  creation  of  a  new  ontology  was  not  very  difficult.  It  took  about  32  man-hours 
to  create  a  paper-based  version  for  the  aircraft  tire  manual  of  49  pages.  With  a  little 
testing  support  of  IS  LA  it  turned  out  to  be  not  very  difficult  to  implement  the  new 
domain  ontology  into  a  CML  file,  and  load  this  in  the  DAS  file  for  the  new 
domain. 

How  difficult  is  it  to  implement  a  (new)  domain  ontology  in  the  tools? 

It  is  rather  surprising,  however,  that  the  manual  manner  of  implementing  and 
building  an  ontology  (as  described  above)  is  easier  than  using  the  ontology 
development  support  tool.  This  tool  certainly  lacks,  especially  a  graphical 
representation  view  of  a  complete  ontology.  The  user-interface  (and  user  manual) 
need  extra  attention  in  the  area  of  usability  (it  takes  quite  some  time  to  find  out 
how  to  add  a  new  concept).  After  these  usability  modifications,  adjusting  and 
adding  new  concepts  to  an  existing  ontology  is  expected  to  be  easier  and  less  error 
prone  via  this  tool. 

How  difficult  is  it  to  tune  the  AIDAS  tools  to  new  source  material? 

Usually,  a  domain  specialist  will  do  the  development  of  a  domain  ontology,  and  the 
necessary  time  and  effort  will  depend  on  the  complexity  and  size  of  the  new 
domain  (amount  of  concepts,  keywords  and  relationships).  However,  our 
expectation  is  that  time  spent  on  ontology  development  will  after  an  initial  “peak” 
of  necessary  hours,  require  only  relatively  few  hours  to  maintain  and  adjust  the 
domain  ontology  when  changes  in  the  domain  occur.  The  real  challenge  is  to 
determine  the  correct  level  of  detail  in  the  domain  ontology,  enabling  optimal 
retrieval  of  fragments.  In  essence  this  comes  down  to  establish  a  return  on 
investment  on  the  amount  of  time  (and  money)  spent  versus  the  attained  level  of 
detail  in  a  domain  ontology  enabling  fragment  indexing,  search  and  retrieval. 
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How  difficult  is  it  to  get  the  text  and  image  analysis  working  in  a  new  domain? 

We  learned  from  the  new  domain  implementation  that  the  document  analysis  (both 
text  and  pictures)  still  requires  some  fine-tuning  to  be  able  to  get  correct  results 
(fragments)  from  the  structure  and  text  analysis.  The  picture  analysis  remains 
problematic,  probably  due  to  the  extensive  grey  scales  of  the  pictures.  The  domain 
ontology  is  loaded  without  problems  for  the  new  domain  context. 

System-based  versus  task-based  manuals 

Very  often,  technical  manuals  for  system  maintenance  are  written  from  a  system- 
oriented  perspective  instead  of  a  task-oriented  perspective.  Although  the  latter 
situation  is  more  ideal  from  an  instructional  point  of  view,  it  is  harder  to  create  an 
ontology  on  the  basis  of  such  a  manual  because  the  table  of  contents  is  not  directly 
suitable  to  use  as  means  of  input  for  domain  ontology  development.  However,  to 
support  searching  in  a  maintenance  context,  it  seems  desirable  to  enhance 
fragments  with  information  on  how  they  are  used  in  a  maintenance  context.  These 
annotations  can  be  added  manually  using  the  instructional  ontology.  This  process 
would  be  supported  by  an  index  of  fragments  on  maintenance  task  involved.  For 
this  purpose,  it  is  interesting  to  study  the  possibility  of  enhancing  the  domain 
ontology  with  a  task-part,  so  that  an  additional  task  index  can  be  generated. 
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4.  Relevant  RNLAF  factors  for  the  use  of  IMAT 

This  chapter  gives  an  overview  of  the  first  part  of  the  factor  study,  i.e.  an  inventory 
of  educational,  technical  and  organisational  factors,  in  relation  to  the  possible  use 
of  IMAT  solutions  in  the  RNLAF.  Chapter  5  describes  the  second  part  of  this 
factor  study,  which  consists  of  the  development  of  factor  models.  Models  are 
developed  to  get  a  clear  view  of  relationships  between  relevant  factors  and  their 
influence  on  the  technical  training  process  with  use  of  IMAT  solutions  in  the 
RNLAF. 

The  factors  have  been  generated  by  means  of  desk-research.  Input  for  this  study 
was  a  literature  study  (See:  list  of  references),  project  members’  knowledge  of  the 
RNLAF  through  TNO  research  projects  for  the  RNLAF  in  the  field  of  training  and 
education,  such  as  IETM,  IMAT,  MTTP  and  e-leaming.  Furthermore, 
organisational  knowledge  on  the  RNLAF  was  gathered  through  a  special  RNLAF 
course  (duration  of  six  days)  for  TNO  researchers.  Finally,  a  symposium  on  the 
RNLAF  educational  policy  for  the  (nearby)  future  was  attended. 


4.1  Context  Description 

Within  an  organisation  as  the  RNLAF  strong  demands  are  made  upon  the  quality 
of  training  and  education.  Training  and  education,  especially  in  the  field  of 
technical  training,  comprises  the  training  of  crucial,  critical  and  often  safety  risk 
skills,  knowledge  and  attitudes. 

For  an  organisation  such  as  the  RNLAF  it  is  crucial  to  determine  the  advantages 
and  risks  of  a  full-scale  implementation  of  IMAT  tools  in  a  new  domain,  especially 
since  the  benefits  of  IMAT  tools  lie  in  technical  domains  with  huge  amounts  of 
(changing)  technical  content.  Therefore  it  is  important  to  know  how  the  IMAT 
tools  can  be  used  by  the  RNLAF  and  which  implications,  guidelines  and  advice 
concerning  the  use  of  IMAT  tools  can  be  derived  from  the  research  in  the  IMAT 
and  IMAT  follow-up  project.  First  thing  to  know  is  how  the  IMAT  Pfinal^  tools 
work  and  what  needs  to  be  done  for  a  new  domain  implementation  in  the  RNLAF. 
This  was  described  in  the  previous  chapter. 

A  second  step  is  to  determine  and  study  relevant  factors  affecting  and  involving  the 
use  of  IMAT  tools  in  the  RNLAF  context.  However  an  important  distinction  can  be 
made  between  the  IMAT  tools  as  they  are  now,  or  ideal  IMAT  tools.  Since  the 
European  IMAT  project  delivered  an  advanced  set  of  prototypes,  methods  and 
knowledge  on  the  integration  of  manuals  and  training,  it  is  important  not  to  limit 
the  discussion  on  the  possible  use  of  IMAT  tools  as  they  are  now.  For  future  use 


*  Pfinal  is  used  as  description  for  the  final  set  of  tools  developed  in  the  IMAT1 
project. 
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within  the  RNLAF  it  is  worthwhile  to  look  beyond  the  possibilities  of  the  current 
prototypes  and  take  also  an  ideal  set  of  IMAT  tools  or  IMAT  kind  of  solutions  into 
account. 


4.2  Factor  Overview 

Any  organisation  considering  the  use  of  technology  in  training,  either  to  design  and 
develop  training  or  to  transfer  training,  will  have  to  think  about  the  consequences 
and  relevant  factors  of  putting  the  technology  into  best  use.  In  our  view  three  main 
perspectives  have  to  be  taken  into  account.  They  are  education,  technology  and 
organisation  (ETO).  By  this  we  mean  that  from  these  three  perspectives  one  has  to 
think  about  the  factors  affecting  and  involving  the  use  of  advanced  technology  in 
training.  The  next  figure  gives  a  graphical  representation  of  the  ETO  framework  to 
study  the  use  of  IMAT  tools  and  solutions  for  the  RNLAF' s  technical  training 
domains. 


Figure  4. 1:  ETO  framework  for  the  study  of  factors  involving  the  use  of  IMAT  tools  for 
technical  training  in  the  RNLAF 

Further  analysis  was  necessary  to  determine  which  factors  play  an  important  role 
from  each  ETO  perspective.  The  results  of  this  analysis  are  described  in  the  next 
sections,  and  from  each  ETO  perspective  relevant  factors  for  the  use  of  IMAT  tools 
in  the  RNLAF  context  are  listed. 


4.3  Educational  Factors 

In  the  RNLAF,  from  an  educational  point  of  view,  many  interesting  developments 
are  taking  place  in  the  field  of  technical  training.  For  the  RNLAF,  training  and 
education  are  keystones  of  their  organisation.  Although  this  could  apply  to  any 


TNO  report 


TNO-DV1  2004  A169 


47 


high  tech  organisation,  for  the  Netherlands  Armed  Forces  (Royal  Netherlands 
Army,  Royal  Netherlands  Navy,  Royal  Netherlands  Airforce  and  the  Royal 
Netherlands  Military  Police)  this  goes  even  further:  their  key  mission  during 
peacetime  is  training  and  education.  To  be  able  to  participate  in  crises, 
peacekeeping  missions  and  times  of  war,  military  organisations  need  constant 
preparation,  training  and  education  to  comply  with  the  high  demands  made  by 
military  operations  throughout  the  world. 

4.3.1  Technical  training  characteristics 

In  general,  technical  training  can  be  defined  by  certain  characteristics  (Barnard,  et 
al.,  2002).  This  section  gives  an  overview  of  these  characteristics,  since  they  also 
apply  to  the  RNLAF.  The  main  characteristics  and  problems  in  technical  training 
are: 

•  Training  is  aimed  both  at  the  acquisition  of  procedural  knowledge  and  skills  as 
well  as  complex  cognitive  skills,  such  as  troubleshooting. 

•  Technical  training  is  often  related  to  crucial,  critical,  and  risky  or  hazardous 
work  (such  as  aircraft  maintenance)  and  is  therefore  very  organisation  and 
equipment  specific. 

•  Training  is  a  mix  of  theory  and  practice,  and  should  preferably  take  place  at  or 
near  the  maintenance  site,  such  as  the  workshop  or  maintenance  hangar. 

•  Training  needs  to  be  accurate  and  up-to-date:  this  is  a  challenge  since  technical 
systems  and  their  components  change  very  rapidly. 

•  There  is  no  training  material  available  on  the  market;  companies  need  to 
develop  and  maintain  their  own  training  material. 

•  Training  developers  and  instructors  are  often  technical  experts  with  insufficient 
didactic  expertise. 

•  Technical  specialists  need  constant  training  and  re-training  because  of  rapidly 
changing  technologies. 

•  Technical  training  courses  usually  have  a  very  limited  number  of  trainees. 

4.3.2  Centralisation  of  training 

A  few  years  ago,  the  RNLAF  decided  to  centralise  most  of  the  training  activities, 
such  as  basic  military  training,  initial  pilot  training,  and  initial  function  related 
technical  training,  such  as  aircraft  maintenance  at  the  Royal  Military  School  (Dutch 
abbreviation:  KMSL),  located  at  the  airbase  of  Woensdrecht.  The  centralisation  of 
training  has  led  towards  an  increasing  number  of  students  on  one  hand,  and  a 
shortage  of  instructors  at  the  Royal  Military  Airforce  School  on  the  other. 
Therefore,  the  RNLAF  is  currently  reviewing  various  options,  such  as  e-leaming, 
training  on  the  job,  and  outsourcing,  to  create  and  maintain  an  efficient  and 
effective  training  program  for  technical  specialists. 

The  IMAT  tools  can  be  used  in  different  ways  by  an  organisation.  Therefore, 
several  scenarios  on  the  possible  use  of  IMAT  will  be  developed  for  the  RNLAF. 
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This  will  be  described  in  the  second  IMAT  report  (Janssen,  et.  al.,  2004).  The 
IMAT  tools  can,  for  instance,  be  used  at  one  central  location  where  training 
material  is  developed.  This  training  material  can  then  be  distributed  to  instructors 
and  students  at  various  locations  within  the  RNLAF. 

4.3.3  Leerbedrijven9 

The  RNLAF  will  start  with  the  implementation  of  so-called  “leerbedrijven”  at 
various  bases  of  the  RNLAF  for  all  (function  related)  technical  training,  apart  from 
the  initial  (function  related)  technical  training  at  the  KMSL  in  Woensdrecht.  These 
schools  will  act  as  intermediaries  between  the  initial  technical  training  at  the 
KMSL  and  training  on  the  job  at  the  operational  airbases. 

4.3.4  Training  on  the  job 

After  initial  training  at  the  KMSL,  technical  specialists  receive  further  training  on 
the  job,  under  guidance  of  a  master,  at  the  squadron  of  the  assigned  airbase.  This 
results  in  a  heavy  burden  on  the  maintenance  capacities  of  an  operational  squadron, 
due  to  the  shortage  of  sufficient  experienced  staff  to  fill  in  both  the  demands  made 
by  the  training  on  the  job  of  novices  and  maintenance  activities  requiring  a  high 
level  of  expertise.  The  IMAT  tools  could  be  used  in  "a  training  on  the  job 
scenario",  where  demands  on  the  master  could  become  less  if  specialists  could  also 
use  self-study  learning  materials,  created  for  instance  with  the  IMAT  tools. 

4.3.5  HI  VO 10 

HIVO  is  a  Dutch  acronym  for  the  project  that  currently  reviews  all  the  initial 
function-related  technical  training  of  the  RNLAF  in  order  to  develop  a  more 
congruent  curriculum  for  technical  specialists.  Until  now,  the  curriculum  for 
technical  specialists  comprised  a  broad  and  extensive  theoretical  training  at  the 
KMSL,  combined  with  practical  training  on  live  mock-ups  of  fighter  jets  and 
engines.  For  some  disciplines,  the  KMSL  also  uses  advanced  simulation-based 
training  methods.  After  the  initial  training,  specialists  receive  further  training  on 
the  job  at  the  assigned  airbase. 


^  This  is  a  Dutch  name  for  schools  located  at  the  airbase  for  (function  related) 
technical  training. 

10  HIVO:  Herziening  Initiele  Vaktechnische  Opleidingen. 
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Problems  identified  with  the  current  curriculum  for  technical  specialists  are: 

•  Strong  fluctuation  in  instructor  capacity  at  the  KMSL  (varying  from  under  to 
overcapacity); 

•  Too  great  a  distance  and  differences  between  training  in  the  school  and  training 
on  the  job; 

•  Technical  specialists  receiving  longer,  extensive  and  broader  initial  functional 
training  than  needed  for  their  initial  functions; 

•  Shortage  of  masters  at  the  operational  airbases. 

Although  there  is  not  a  direct  link  between  the  HIVO  project  and  the  IMAT  follow¬ 
up  research,  it  is  good  to  realise  and  understand  the  background  factors  underlying 
the  HIVO  project.  They  can  hold  important  implications  for  the  use  of  IMAT  tools 
within  the  RNLAF.  However,  results  from  this  project  were  not  available  during 
the  IMAT  follow-up  research. 

4.3.6  Simulation-based  training 

The  RNLAF  uses  both  high  fidelity  simulations  and  medium  fidelity  type 
simulations  (Jacobs,  van  der  Hulst  &  van  der  Stigchel,  2002)  for  the  instruction  of 
technical  specialists.  At  the  Royal  Military  School,  for  example,  (live)  mock-ups  of 
F-16  fighter  jets  and  F-16  engines  are  used  in  a  practice  environment  to  learn  basic 
maintenance  skills.  For  avionics  specialists  in  training  for  F-16  maintenance,  in  an 
advanced  medium  fidelity  type  simulation-based  training  environment,  the  so- 
called  Avionics  Technical  Training  Package  (ATTP)  is  located  at  the  school’s 
premises.  A  similar  simulation-based  training  environment,  the  Mechanical 
Technical  Training  Package  (MTTP),  is  currently  under  development  for  F-16 
system  maintenance  specialists.  The  RNLAF  conducts  the  development  and 
procurement  of  both  training  packages  in  a  partnership  with  the  Norwegian 
Airforce.77 

At  the  main  operating  bases  of  the  RNLAF,  so  called  Unit  Level  Trainers  (ULT) 
are  located  at  the  squadrons.  These  high  fidelity  type  simulations  are  not  only  used 
for  F-16  pilot  training  but  also  for  the  training  and  licensing  of  technical  specialists, 
who  have  to  conduct  maintenance  activities  in  the  cockpit  of  the  F-16. 

Although  the  use  of  simulations  holds  many  advantages  for  the  effective  and 
efficient  training  of  technical  specialists,  the  RNLAF  currently  faces  many 
challenges  in  keeping  the  training  materials  of  their  simulation  based  environments 
up  to  date  with  the  changes  in  the  core  systems  of  the  F-16  and  their  technical 


77  Several  European  Airforces  (EPAF)  have  joined  in  a  “F-16  partnership”  to  work 
with  each  other  on  matters  of  procurement,  logistics,  maintenance  and  training. 
Rationale  behind  this  partnership  can  be  summarised  as  forming  a  strategic 
alliance,  and  achieving  mutual  adjustments  in  logistics,  maintenance  and  training. 
This  leads  to  better  and  closer  co-operation  during  NATO  missions,  e.g.  enabling 
Dutch  technicians  to  work  on  Belgium  F-16’s  and  vice  versa.  EPAF  partners  are 
Belgium,  Denmark,  Norway,  Portugal  and  The  Netherlands. 
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manuals.  Since  the  source  material  for  these  training  simulations  are  regularly 
changing  and  updated  technical  manuals,  the  IMAT  tools  could  provide  a  possible 
solution  in  keeping  the  training  material  for  simulations  accurate  and  up  to  date  in  a 
better  and  easier  way. 

4.3.7  E-learning 

Within  the  RNLAF  e-leaming  is  not  necessarily  seen  as  a  better  way  of  transferring 
knowledge,  but  seen  as  a  possible  solution  towards  the  effective  transfer  of  training 
as  well  as  an  opportunity  to  provide  training  modules  which  can  be  studied  by 
students  anywhere  and  anytime.  For  instance,  e-leaming  could  facilitate 
(re)training  on  the  job. 

E-leaming  is  an  educational  technology  closely  connected  to  the  possible  use  of 
training  material  created  with  the  IMAT  tools  and  is  an  important  educational 
factor  to  take  into  account  when  thinking  on  future  IMAT  solutions  for  the 
RNLAF. 

4.3.8  ADL-SCORM 

The  RNLAF  decided  to  follow  the  ADL  SCORM  guidelines  for  the  standardisation 
of  e-leaming  content  (Hylkema,  2001).  It  is  therefore  important  for  the  RNLAF  to 
know  whether  the  IMAT  tools  can  deliver  ADL  SCORM  compliant  e-leaming 
content.  The  ontologies  used  in  IMAT  for  the  indexing  of  fragments,  or  in  other 
words  to  add  metadata,  were  developed  with  a  close  eye  on  the  work  done  in 
projects  as  Ariadne,  IMS,  LOM  and  ADL-SCORM. 

An  IMAT/  ADL-SCORM  investigation  by  TNO  (Veerman,  et  al.,  2002)  showed 
that  differences  exist  in  the  naming  of  metadata  fields  and  also  in  the  type  of  fields. 
The  IMAT  didactic  indexing,  for  example,  is  far  more  extensive  than  the  didactic 
indexing  provided  by  SCORM.  Both  SCORM  and  IMAT  use  XML,  and  although 
IMAT  XML  files  cannot  be  used  directly  by  SCORM,  the  database  structure  seems 
appropriate  to  create  a  link  with  SCORM. 

To  create  SCORM-compliant  IMAT  solutions,  obligatory  and  missing  SCORM 
metadata  fields  should  be  filled  in  by  IMAT.  By  using  mapping  techniques  and 
creating  profiles  this  problem  can  be  addressed.  The  extra  metadata  provided  by 
IMAT  can  also  be  used.  The  metadata  fields  that  are  obligatory  by  SCORM  should 
be  made  available  via  an  extra  (IMAT)  module  that  should  be  developed.  This 
extra  module  can  provide  an  exchange  platform  for  fragments  and  metadata 
through  a  SCORM  XML  connection.  For  a  future  existence  of  IMAT  in  the 
RNLAF  such  an  exchange  platform  is  essential.  The  advantage  of  developing  an 
extra  IMAT/SCORM  XML  module  is  that  in  the  future  only  this  module  will  have 
to  be  adjusted  in  case  XML  specifications  change.  See  Veerman,  Veldhuis,  Jacobs 
&  Strijker  (2002). 
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4.3.9  Outsourcing 

Training  outsourcing  could  be  applied  to: 

•  Technical  training  as  a  whole:  Students  would  be  trained  outside  the  company 
or  an  external  organisation  would  provide  training  within  the  company. 

•  Technical  training  development:  For  the  development  of  advanced  training 
solutions  such  as  computer-based  training  (CBT),  simulation  and  web-based 
training  ( WBT)  one  may  consider  outsourcing  the  development  function  to 
respond  to  a  possible  lack  of  expertise. 

Outsourcing  of  training  can  provide  an  alternative  for  instructor  capacity  problems 
at  the  Royal  Military  School;  though  this  alternative  certainly  cannot  be  applied  to 
all  technical  training  that  is  very  specific  and  closely  adapted  to  the  systems  in  use 
by  an  organisation.  In  the  Netherlands,  F-16  configurations,  for  instance,  differ 
substantially  from  F-16  configurations  in  other  countries.  Also  differences  between 
the  organisation  of  maintenance  work  in  the  RNLAF  and  foreign  airforces 
sometimes  prevent  outsourcing  being  a  good  alternative.  Finally,  the  military 
saying  "today's  friend  can  be  tomorrow’ s  enemy"  suggests  certain  risks  in 
outsourcing  critical  and  specific  technical  training. 

Outsourcing  of  training  development  could  provide  a  solution  to  fill  any  gap  in 
expertise;  though  this  will  not  necesserily  mean  that  it  is  cheaper  or  better  for  an 
organisation  such  as  the  RNLAF  to  follow  this  path.  Especially  in  the  field  of 
technical  training,  extensive  domain-related  knowledge  is  required.  Outsourcing 
training  development  would  still  require  a  substantial  contribution  of  RNLAF’s 
technical  and  domain  expertise.  This  was,  for  instance,  the  case  during  the 
development  of  the  ATTP.  Furthermore,  the  same  arguments  as  described  in  the 
previous  section  can  be  made  against  the  outsourcing  of  training  development 
regarding  the  specificity  and  confidentiality  of  technical  systems  and  their  system 
knowledge.  Finally,  in  technical  training,  material  needs  constant  revision  to  keep 
it  up  to  date  and  accurate.  This  implies  that  a  certain  amount  of  training 
(re)development  always  will  always  have  to  take  place  within  the  RNLAF. 

For  the  RNLAF,  it  is  therefore  important  to  have  tools  available  supporting 
effective  and  efficient  training  development.  This  is  especially  the  case  for 
technical  training  and  the  development  of  advanced  training  solutions  such  as 
simulation-based  training  and  e-leaming.  The  main  objective  of  the  IMAT  tools  is 
to  provide  automated  and  structured  support  in  the  process  of  creating  training 
material  from  technical  documentation. 
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4.3.10  Summary 

The  previous  section  described  these  educational  factors: 

•  Technical  training  characteristics:  important  characteristics  for  technical 
training  and  possible  use  of  IMAT  tools. 

•  Centralisation  of  initial  technical  training:  at  the  KMSL  in  Woensdrecht. 

•  Leerbedrijven:  schools  to  be  founded  at  airbases  in  the  near  future. 

•  Training  on  the  job:  important  focus  for  technical  training  in  the  RNLAF. 

•  E-leaming:  an  important  factor  for  technical  training  in  the  RNLAF  closely 
related  to  the  possible  use  of  IMAT  solutions. 

•  Simulation-Based  Training:  an  important  training  mechanism  for  technical 
training  in  the  RNLAF  closely  related  to  the  possible  use  of  IMAT  solutions. 

•  HIVO:  a  project  that  reviews  all  initial  function-related  technical  training. 

•  Standards:  ADL-SCORM,  adapted  by  the  RNLAF  for  standardisation  of 
metadata  for  electronic  content. 

•  Outsourcing:  possible  solution  for  technical  training  and/or  technical  training 
development  with  important  consequences  for  the  RNLAF  and  possible  use  of 
IMAT  solutions. 

These  educational  factors  are  reviewed  further  by  the  development  of  factor  models 
in  the  next  chapter.  The  development  of  models  assists  the  process  of  determining 
the  relative  importance  of  factors  for  the  RNLAF,  relationships  between  factors 
and  their  influence  on  the  possible  use  of  IMAT  solutions.  The  factor  models  give 
a  clear  view  of  the  relationships  between  educational  and  other  factors.  The  models 
also  enhance  knowledge  on  how  and  where  the  factors  influence  the  technical 
training  process  in  the  RNLAF  with  use  of  IMAT  tools. 


4.4  Technology  Factors 

Technical  factors  are  a  basic  condition  for  success  or  failure  of  a  technology-based 
training  solution.  Failing  technology  will  prevent  successful  implementation  of 
technology-based  training  solutions  in  an  organisation,  especially  on  a  large  scale. 
So  it  is  important  to  identify  and  quantify  relevant  technology  factors  and  their 
influence  on  the  possible  use  of  IMAT  in  the  RNLAF. 

4.4.1  Reliability 

Any  training  solution  involving  technology  should  address  the  issue  of  reliability. 
By  reliability  is  meant  the  technical  reliability  of  tools  and  technical  solutions.  This 
is  a  important  factor  in  the  use  of  IMAT  tools  as  huge  amounts  of  technical 
documentation  are  the  basis  for  the  development  of  (electronic)  training  material. 
Reliable  technology  is  a  prerequisite  to  be  able  to  work  with  IMAT  tools. 
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In  the  IMAT  context  it  is  crucial  to  have  a  document  analysis  tool  such  as  AIDAS, 
which  is  fully  reliable  in  creating  and  retrieving  fragments.  A  second  important 
factor  is  the  reliability  of  the  database  and  the  way  data  is  handled  and  stored.  A 
corrupt  database  would  be  a  nightmare,  especially  in  large  and  extensive  technical 
domains. 

4.4.2  Reusability 

The  concept  of  reusability  was  central  in  the  definition  of  the  IMAT  project  and  is 
crucial  to  determine  the  usability  of  the  IMAT  tools.  Within  the  IMAT  context 
reusability  can  be  defined  in  several  ways.  For  example:  the  reusability  of 
fragments,  instructional  bags  and  annotations,  but  also  the  reusability  of  ontologies 
and  knowledge.  The  latter  two  are  examples  of  reusability  of  concepts  and 
knowledge.  The  first  type  of  reusability,  concerning  the  reuse  of  fragments  in 
various  lifestages,  is  more  technical  in  nature  and  requires  version  management  and 
a  (semi)-automatic  IMAT  tool  change  and  update  process  that  will  enable  the  reuse 
of  content. 

For  the  RNLAF,  it  is  essential  to  determine  how  the  reusability  provided  by  the 
IMAT  tools  answers  problems  in  training  development  for  technical  domains  and 
to  determine  which  improvements  on  this  matter  within  the  IMAT  tools  are 
necessary  or  desirable. 

4.4.3  Technology  level 

The  use  of  advanced  technology  in  training  also  means  that  one  has  to  consider  the 
level  of  technology  competence  of  the  organisation.  By  this  we  imply  that,  both  at 
the  organisational  level  and  at  individual  level,  a  careful  analysis  has  to  be  made 
whether  or  not  an  organisation  is  capable  of  implementing  technology-based 
solutions.  The  analysis  must  also  include  an  inventory  of  actions  to  be  taken,  such 
as  end-user  training,  reorganisation  of  infrastructure,  or  innovation  and  change 
management  to  come  to  a  successful  implementation.  This  is  especially  the  case  for 
large-scale  technology  solutions  such  as  e-leaming  or  IETM  implementation. 

For  the  use  of  IMAT  tools  in  the  RNLAF,  it  is  advisable  to  make  at  least  a  quick 
scan  of  the  technology  level  of  involved  organisation  parts  and  the  technology  level 
of  involved  people  in  various  functions  and  roles  (students,  developers,  instructors, 
management,  etc.). 

4.4.4  Infrastructure 

Some  years  ago  the  RNLAF  created  a  confidential  network  connecting  all  locations 
within  the  Netherlands  abbreviated  in  Dutch  as  KLUIM.  With  the  implementation 
of  KLUIM  so  called  KLUIM  desktop  configurations  were  installed  following  the 
LAN  2000  standard  and  KLUIM  standard.  The  LAN  2000  standard  ensures  that 
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each  desktop  computer  and  laptop  configuration  is,  to  a  large  extent,  the  same.  The 
KLUIM  standard  enables  the  connection  of  LAN  2000  computers  in  a  single  and 
large  secured  network.  Through  this  network  (an  intranet)  it  is  possible  to  exchange 
information  and  messages  in  a  secure  way. 

Through  the  implementation  of  the  KLUIM  and  LAN  2000  standard,  the  personal 
computer  has  become  more  and  more  a  company  computer.  The  organisation 
determines  which  software  is  required  in  certain  functions  and  how  the 
configuration  of  a  computer  is  set  up.  One  advantage  is  that  new  software  releases 
and  updates  can  be  done  automatically  by  the  ICT  maintenance  organisation  at  a 
distance.  The  Defence  Telematics  Organisation  (DTO)  is  the  organisation 
responsible  for  ICT  maintenance,  software  releases,  change  and  updates  and  the 
security  of  the  network  for  the  entire  Netherlands  Armed  Forces. 

For  end-users,  the  implementation  of  KLUIM  means  that  there  is  no  longer  a  need 
for  their  own  PC.  All  data  is  stored  at  the  centre  and  users  are  able  to  retrieve  this 
data  after  logging  in  on  any  KLUIM  PC.  Back-up  and  storage  of  data  are  made 
automatically  by  the  system  managers.  Disadvantages  for  end-users  are  that  it  is 
impossible  to  install  software  individually  and  provide  Internet  acces  from  these 
PCs.  The  latter,  however,  can  be  done  from  separate  stand-alone  machines. 

Other  matters  relevant  to  the  KLUIM  network  include  the  fact  that  security  rules 
and  high  costs  of  installation,  change  and  updating  by  DTO  of  (new)  software 
releases  have  prohibited,  so  far,  the  use  of  advanced  training  solutions  such  as  the 
ATTP  simulation-based  training  over  various  locations. 

The  features  of  a  confidential  network  can  lead  to  consequences  for  successful 
implementation  of  a  technology-based  training  solution  such  as  IMAT,  and  need  to 
be  closely  studied  and  defined,  especially  in  scenarios  considering  a  network¬ 
centric  solution  for  the  use  of  IMAT  tools. 

4.4.5  Data  Security 

Although  this  is  closely  related  to  the  previous  factor,  it  is  important  to  realise  that 
data  security  is  necessary  due  to  confidentiality  and  classification^2  of  source 
material  (such  as  technical  manuals  or  training  material).  Although  the  RNLAF 
network  is  a  confidential  network  (an  intranet  with  datasecurity)  classified 
information  is  not  allowed  on  this  network. 

In  the  case  of  working  with  IMAT  tools  outside  the  RNLAF  network,  for  example 
online,  wireless  or  at  locations  outside  The  Netherlands,  datasecurity  will  become 


Classification  of  source  material  is  in  the  RNLAF  divided  over  various  classes.  In 
technical  domains  source  material  is  very  often  “business  confidential”.  In  the 
RNLAF  certain  information  is  classified,  resulting  in  stricter  regulations 
concerning  the  distributions  and  use  of  this  material. 


12 


TNO  report 


TNO-DV1  2004  A169 


55 


an  issue,  as  well  as  in  The  Netherlands  when  classified  material  is  used  and  needs 
to  be  distributed  (electronically). 

Consequences  for  the  use  of  IMAT  will  depend  on  the  classification  of  the  source 
material  and  whether  training  material  needs  to  be  distributed  outside  the  RNLAF 
network  though  this  does  not  seem  very  likely.  An  interesting  question  is  when 
data  security  is  necessary,  whether  data  security  should  be  integrated  in  an  IMAT 
solution  or  should  be  addressed  in  the  distribution  process. 

4.4.6  Summary 

The  previous  section  described  these  technical  factors: 

•  Reliability:  technical  reliability  of  IMAT  tools. 

•  Reusability:  reusability  of  fragments  or  electronic  content  in  general. 

•  Technology  level:  a  factor  to  take  into  account  in  the  implementation  of  IMAT 
tools  in  a  certain  part  of  the  RNLAF  organisation. 

•  Infrastructure:  an  important  factor  to  take  into  account  in  the  implementation  of 
IMAT  tools. 

•  Data  Security:  an  important  factor  in  case  source  material  is  classified. 

These  technical  factors  are  studied  further,  together  with  educational  and 
organisational  factors  through  the  development  of  factor  models,  in  the  next 
chapter. 


4.5  Organisational  Factors 

Many  organisational  factors  (at  various  levels)  have  to  be  taken  into  account  when 
an  organisation  considers  the  use  and  implementation  of  a  technology-based 
training  solution  such  as  IMAT.  Although  this  is  no  different  from  more  traditional 
training  solutions,  the  use  of  technology  can  have  other  implications  for  an 
organisation  such  as  described  in  the  previous  and  current  section. 

4.5.1  Airpower 

Airpower  is  the  “main  product”  of  the  RNLAF  and  can  be  seen  as  the  central 
organisational  mission  statement  from  an  organisational  point  of  view.  Although 
there  is  no  direct  link  connecting  the  use  of  IMAT  tools  with  this  central  mission, 
training  and  training  development  solutions  such  as  IMAT  will  all  lead  to  an 
effective  and  qualified  organisation  able  to  fulfil  this  key  mission.  Seen  from  this 
perspective  the  key  mission  of  airpower  is  also  closely  related  to  the  technical 
(training)  domain,  for  which  the  use  of  IMAT  tools  is  intended. 
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4.5.2  Defence  organisation 

It  may  appear  obvious  that  the  RNLAF  is  a  military  organisation.  However,  it 
certainly  is  not  a  factor  to  forget  especially  when  training  in  technical  domains  is 
considered  as  well  as  the  impact  of  using  technology  in  training.  Two  examples 
clarify  the  importance  of  this  factor: 

•  Distributed  learning  solutions  (such  as  e-leaming),  and 

•  Confidentiality  of  learning  content. 

Distributed  and  advanced  learning  solutions  within  the  RNLAF  are  limited  in  full- 
scale  implementation  due  to  regulations  concerning  the  confidential  network  of  the 
RNLAF.  The  ATTP,  for  example,  had  been  intended  for  use  at  the  KMSL  and 
various  bases.  Due  to  regulations  preventing  the  use  of  authoring  tools  and 
distribution  of  “exotic”  formats  such  as  multimedia  formats,  the  ATTP  nowadays  is 
only  used  as  training  package  at  the  KMSL. 

Important  implications  concerning  the  confidentiality  of  learning  content  or 
training  material  were  described  as  technology  factor  data  security  in  section  4.4.5. 

4.5.3  Aircraft  maintenance 

The  largest  technical  training  domain  within  the  RNLAF  concerns  aircraft 
maintenance  for  fighter  jets  (F- 16),  transportation  planes,  and  helicopters.  This 
fact,  combined  with  very  interesting  developments  in  this  domain,  such  as  the 
internationalisation  of  standards  for  aircraft  maintenance  and  the  implementation  of 
(I)ETMs,  determine  the  decision  to  focus  further  in  this  research  on  aircraft 
maintenance  training  within  the  technical  training  domain. 

4.5.4  Flight  safety/  JAR  &  LE 

For  an  aviation  organisation  such  as  the  RNLAF,  participating  daily  in  missions  all 
over  the  world,  flight  safety  is  a  key  factor.  Flight  safety  starts  on  the  ground  with 
proper  and  thorough  aircraft  maintenance. 

The  RNLAF  follows  European  standards  for  the  civil  aviation  industry,  the  JAR- 
147  for  aircraft  maintenance  and  the  JAR-66  for  technical  aviation  training,  and  an 
own  military  standard  (LE)  derived  from  civil  aviation  regulations  and  military 
characteristics.  The  use  of  weapon  systems,  for  instance,  is  an  important  difference 
between  civil  and  military  aviation  organisations.  Factors  concerning  the 
transportation,  use,  training,  and  maintenance  of  these  weapon  systems  are 
examples  of  extra  rules  and  standards  needed  for  military  aviation  organisations. 

These  standards  not  only  prescribe  tasks,  procedures  and  rules  for  proper  flight 
maintenance  and  qualification  tests  for  aircraft  and  their  systems,  but  also  have  an 
impact  on  the  training  of  maintenance  engineers.  Strong  implications  from  these 
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standardisation  rules  are  the  importance  of  training  (including  qualifications)  and 
the  requirements  and  regulations  concerning  working  with  up  to  date  technical 
documentation.  Following  wrong  procedures  in  “old  documentation”  can  lead  to 
costly  maintenance  repair  faults  or  even  worse,  serious  damage  to  personnel.  An 
ideal  set  of  IM  AT  tools  should  certainly  contain  a  solution  for  keeping  training 
material  up  to  date  and  accurate  in  an  automated  and  easy  way. 

4.5.5  (I)ETM  implementation 

The  evolution  from  paper  manuals  to  electronic  manuals  started  in  the  technical 
maintenance  field  in  the  early  1990s.  Electronic  manuals  offer  many  benefits,  such 
as  rapid  and  easy  centralisation  of  data  management,  conversion  management, 
distribution  and  storage. 

The  American  Department  of  Defence  started  a  standardisation  initiative  (CALS), 
Jorgensen,  1994)  for  the  development  and  classification  of  Electronic  Technical 
Manuals  (ETMs)  or  Interactive  Electronic  Technical  Manuals  (IETMs).  This 
initiative  has  evolved  worldwide  and  led  to  numerous  military  standards  and 
specifications  for  various  classes  of  (I)ETMs.  It  is  nowadays  followed  mainly  by 
military  organisations  and  affiliated  industries  in  technical  systems  and  technical 
documentation. 

In  general  terms,  an  (I)ETM  can  be  described  as  a  package  of  information  used  for 
the  diagnosis  of  technical  (weapon)  systems,  optimally  organised  and  formatted  for 
interactive  end-user  screen  presentations.  An  (I)ETM  can  hold  information  sources 
in  any  format,  varying  from  text,  pictures,  audio  and  video,  in  various  data 
structures  and  data  relationships.  An  (I)ETM  usually  consists  of  maintenance 
procedures,  task  descriptions,  and  structured  help  on  troubleshooting  procedures, 
system  functional  overviews,  and  sometimes  a  training  view  or  mode.  The  five 
(I)ETM  classes  are: 

1 .  Electronic  indexed  pages. 

2.  Electronic  scrolling  documents. 

3.  Linear  structured  ETMs:  content  (index,  text  pictures,  etc.)  presented  in 
separate  frames.  User  navigation  is  possible  through  various  search  options  and 
hyperlinks. 

4.  Hierarchically  structured  IETMs:  based  upon  a  relational  database  for  storage 
of  dynamic  objects.  The  user  interface  is  no  longer  based  upon  frames  and  is 
Windows-based.  For  the  user  the  largest  difference  lies  in  the  user-system 
interactivity  and  the  way  a  user  is  guided  through  steps  in  a  task  or  procedure. 

5.  Integrated  databases  combine  the  functionality  offered  by  class  4  IETMs  with 
opportunities  to  connect  the  IETM  database  directly  with  other  relevant 
databases,  for  instance  a  training  database  or  maintenance  and  part  databases, 
to  create  a  complete  integrated  chain  and  exchange  of  maintenance-related 
information. 
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(I)ETM  classes  are  not  exclusively  defined,  but  generally  speaking,  higher  classes 
provide  extra  functionality  not  present  in  the  lower  classes.  Furthermore,  large 
differences  exist  between  the  underlying  architecture  and  level  of  interactivity 
between  class  1,  2  and  3  ETMs  and  class  4  and  5  IETMs  The  two  highest  classes 
both  have  a  relational  database  for  storage  of  various  data  elements  as  dynamic 
objects.  Also,  class  4  and  5  IETMs  have  intelligent  tables  and  dialog  boxes,  where, 
based  upon  user  prompts  and  user-system  interaction,  the  correct  next  step  in 
maintenance  procedures  are  presented.  This  facility  enables  the  user  to  jump 
directly  towards  the  correct  next  step  in  a  task  instead  of  following  linearly  the 
sequencing  of  a  task.  For  maintenance  it  is  possible  to  insert  so-called  In  Process 
Inspections  (IPIs)  at  desired  locations  in  the  (I)ETM,  to  enable  the  verification  and 
authorisation  by  senior  specialists. 


The  next  figure  gives  a  graphical  presentation  of  the  five  (I)ETM  classes,  as 
presented  by  the  International  Technical  Order  Digitisation  (ITOD).  In  this  figure 
the  two  (I)ETMs  to  be  implemented  for  F-16  maintenance  in  the  RNLAF  in  2003 
and  2004^  are  marked  separately.  The  RNLAF  receives  for  the  on-equipment  F-16 
maintenance  a  class  3  ETM  delivered  by  LM  AERO;  for  the  off-  equipment  F-16 
engine  maintenance  a  class  4  IETM  delivered  by  Pratt  and  Whitney.^ 
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Figure  4.2:  Overview  of  (I)ETM  Classes  with  F-16  (I)ETMs 


13  The  F-16  system  conversion  from  paper-based  manuals  to  (I)IiTMs  will  probably 
be  completed  in  2010.  The  delivery  in  2004  contains  the  F-16  organisational  level 
manuals  only  for  the  F-16  airframe  and  the  intermediate  level  for  the  F-16  engine. 

14  F-16  maintenance  is  divided  in  three  levels:  organisational  (on-equipment 
maintenance),  intermediate  (off-equipment  maintenance)  and  depot  level 
maintenance. 
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Because  of  the  implementation  of  F-16  (I)ETMs,  the  RNLAF  needs  to  review  and 
adapt  F-16  maintenance,  their  maintenance  organisation,  infrastructure  and 
technical  training  for  an  optimal  implementation  and  support  of  the  maintenance 
process  with  “electronic  books”  and  consequently  the  use  of  e-tools  by  technical 
specialists.  See  for  more  information  Riemersma,  Bots,  van  Esch,  Jacobs  &  van  der 
Arend  (2002). 

In  the  RNLAF,  the  first  1ETM  was  introduced  a  few  years  ago  for  the  maintenance 
of  Apache  helicopters.  Although  the  Apache  IETM  is  a  class  5  1ETM,  delivered  by 
Boeing,  certain  problems  regarding  the  implementation  and  organisation  specific 
characteristics,  such  as  the  extensive  use  of  country  peculiar  data  (so  called 
RNLAF  supplements),  prevented  the  RNLAF  from  an  optimal  implementation  and 
use  of  these  IETMs  so  far. 

First  of  all,  many  representation  problems  can  and  have  occurred  in  the  Apache 
IETM  as  a  result  of  limited  presentation  options,  especially  for  schemes  and 
drawings.  The  fact  that  these  are  also  implemented  in  an  electronic  format,  and  can 
only  be  read  from  a  14  or  1 5-inch  computer  or  laptop  screen,  severely  hampers 
instruction  and  maintenance  activities. 

Since  the  RNLAF  holds  no  access  to  the  IETM  database  it  is  not  possible  to  add  or 
alter  material  in  it.  This  has  resulted  in  many  problems  with  Apache  maintenance 
and  training.  In  maintenance,  the  largest  drawback  is  the  fact  that  it  is  not  possible 
to  add  RNLAF  supplements  in  an  electronic  format  to  the  IETM.  This  results  in  “a 
maintenance  support  environment”  for  Apache  specialists  with  the  IETM  installed 
on  a  (ruggedized)  laptop,  where  it  is  obligatory  and  very  often  necessary  to  check 
simultaneously  the  RNLAF  supplements  (see  next  section)  in  paper  format,  leaving 
literally  no  hands  free  to  conduct  maintenance. 

The  RNLAF  uses  Interactive  Multimedia  Instruction  (IMI)  as  CBT  for  technical 
training  of  Apache  specialists  (van  de  Laak  &  Boot,  2001).  The  IMI  uses  IETM 
source  material  but  lacks  for  example  an  automatic  change  and  update  mechanism 
and  does  not  contain  the  latest  version  of  the  Apache  IETM  source  material. 

Other  facilities  provided  by  a  class  5  IETM,  such  as  automatic  changes  and 
updates,  connection  to  a  parts  and  logistics  database  or  training  environment,  have 
not  been  implemented  in  the  RNLAF.  In  essence,  this  is  due  to  a  lack  of  putting  the 
technology  and  maintenance,  training  and  logistics  organisation  in  place,  necessary 
to  implement  the  concept  of  a  class  5  IETM  to  benefit  fully  from  this  solution  as 
maintenance  and  training  organisation.^  Also  it  became  clear  that  what  is 
sometimes  referred  to  as  training  by  (I)ETM  suppliers  certainly  does  not  match 


^  Please  note  that  the  RNLAF  has  30  Apaches,  and  therefore  only  a  small  number 
of  Apache  specialists.  This  makes  concerns  with  a  lack  of  automatic  distribution 
and  updates  far  more  relevant  than  for  the  F-16,  with  hundreds  of  (I)ETM  end- 
users. 
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with  criteria  set  for  proper  training  material  by  instructors  and  instructional 
designers. 

The  implementation  of  F-16  (I)ETMs  and  the  Apache  IETM  clearly  underlines  the 
importance  for  the  RNLAF  of  knowing  and  understanding  the  parallels  and 
differences  between  opportunities  offered  by  IETMs  and  the  IMAT  tools  for 
technical  training  and  maintenance.  This  also  applies  to  the  consequences  for 
technical  training  due  to  the  (I)ETM  implementation.  See  for  more  information: 
Riemersma,  van  Berio  &  Veldhuis  (2002).  (I)ETMs  offer  the  delivery  of  electronic 
source  material,  more  or  less  suitable  as  learning  material,  depending  on  the  type 
and  quality  of  the  technical  documentation.  However,  this  material  is  written  for 
system  maintenance  and  not  always  directly  usable  as  training  material.  The  IMAT 
tools  can  assist  the  training  development  process,  and  will  require  with  (I)ETMs  as 
input  a  more  simple  process  of  document  analysis,  although  this  will  also  largely 
depend  on  the  granularity  (size  of  fragments)  and  type  of  indexing  provided  by  the 
IETMs.  Main  problem  with  the  (I)ETMs  is  that  the  RNLAF  does  not  have  fully 
access  to  the  electronic  content  (stored  in  a  database).  Furthermore,  the  RNLAF,  as 
relative  small  airforce,  has  very  often  no  direct  contract  relationship  with  the 
(I)ETM  suppliers,  and  therefore  no  direct  influence  on  the  type  and  quality  of  the 
(I)ETM.  In  this  matter  the  RNLAF  has  to  depend  largely  on  their  relationship  with 
large  airforces  such  as  the  USAF. 

In  case  source  material  for  technical  training  is  an  (I)ETM,  IMAT  tools  should  be 
able  to  handle  the  data  format  of  the  electronic  content  as  provided.  Proper  access 
to  this  content  is  essential.  For  a  class  4  and  5  IETM  it  is  important  to  determine 
whether  the  training  facilities  provided  by  an  IETM  offer  a  training  solution  and 
how  they  should  be  integrated  in  the  technical  training  curriculum,  or  that  it 
remains  necessary  to  develop  or  procure  (extra)  training  material.  When  using 
IMAT  solutions,  considerable  attention  should  be  paid  to  the  architecture  in 
relation  to  the  RNLAF  infrastructure  and  ICT  policy.  This  should  be  done  in 
combination  with  defining  how  to  support  efficiently  technical  training 
(development),  system  maintenance  and  change  and  update  management,  similar  to 
the  pre-investigation  of  the  (I)ETM  implementation  via  KLUIM  as  anticipated  for 
the  F-16  (I)ETMs. 
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4.5.6  RNLAF  Supplements 

The  RNLAF  uses  (extensively)  so-called  RNLAF  supplements,  especially  in  the 
field  of  aircraft  maintenance.  These  supplements  can  be  described  as  additional 
information  used  along  with  or  instead  of  the  technical  system  and  maintenance 
documentation  provided  by  the  supplier.  An  RNLAF  supplement  is  always  leading 
the  way  as  to  how  system  maintenance  should  be  conducted  and  could,  for 
instance,  describe  temporary  procedures  and  task  descriptions  (written  to 
compensate  for  mistakes  in  the  technical  manuals),  country  peculiar  maintenance 
procedures  and  safety  rules. 

The  RNLAF  relies  heavily  upon  the  use  of  RNLAF  supplements  to  conduct  aircraft 
maintenance,  due  to  time-consuming  procedures  for  new  releases  of  technical 
manuals,  concerning  changes  and  updates  on  mistakes  or  new  procedures.  Also, 
differences  between  law  and  regulations  in  various  countries  (for  instance  on  the 
use  of  tooling,  equipment  and  substances)  and  something  which  could  probably  be 
described  as  “organisational  knowledge  on  systems”  leads  to  the  extensive  use  of 
this  type  of  information.  The  expectation  is  that  with  the  ageing  of  weapon  systems 
the  use  of  RNLAF  supplements  will  not  only  remain  an  important  factor  in  system 
maintenance  (and  training)  but  will  also  increase  in  importance. 

When  RNLAF  supplements  are  part  of  the  training  material,  IMAT  solutions 
should  be  able  to  handle  this  type  of  information.  Currently,  the  RNLAF 
supplements  for  the  Apache  and  F-16  are  created  in  PDF  format.  The  class  4  IETM 
concerning  the  F-16  engine  maintenance  provides  a  connection  with  external 
documents  in  PDF  format  through  hyperlinks.  A  reference  to  the  hyperlink  is 
stored  in  the  IETM  database,  enabling  F-16  countries  to  add  country  peculiar  data 
to  the  IETM  via  a  hyperlink. 

4.5.7  Availability  of  personnel 

In  technical  domains  in  general  and  aircraft  maintenance  in  particular  availability 
of  maintenance  personnel  is  a  serious  problem.  Due  to  a  strong  economy  in  the 
1990s  and  an  undercapacity  of  technical  specialists  on  the  job  market  in  the 
Netherlands,  it  has  been  very  difficult  for  technical  organisations  such  as  the 
RNLAF  to  recruit  and  keep  technical  specialists.  As  a  consequence,  the  RNLAF  is 
confronted  with  a  key  staff  shortage,  resulting  in  a  triple  capacity  problem  in  the 
technical  training  domain.  There  is  a  shortage  of: 

•  Instructors  at  the  Royal  Military  School, 

•  Training  developers,  especially  CBT  or  WBT  developers, 

•  Masters  to  guide  the  training  on  the  job  of  novices. 

In  all  situations,  experienced  technical  specialists  are  required  to  do  the  job. 
However,  their  expertise  is  also  needed  (even  more  so)  for  highly  specialised  and 
difficult  maintenance  tasks,  requiring  a  lot  of  experience,  such  as  troubleshooting. 
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It  is  without  doubt  the  largest  challenge  for  the  RNLAF  to  meet  both  demands 
made  by  operational  maintenance  work  and  the  technical  training  necessary  to 
conduct  maintenance  according  to  the  standards  and  qualifications  on  flight  safety, 
and  aircraft  maintenance.  Both  operational  work  and  training  rely  heavily  upon  the 
scarce  trained  and  experienced  staff  and  both  worlds  are  equally  important  for  the 
daily  and  future  operations  of  the  RNLAF. 

The  use  of  IM  AT  tools  alone  cannot  lead  to  a  solution  for  the  challenges  described 
above.  However,  they  could  make  a  major  contribution  to  the  problem  of  how  to 
support  scarce  expertise  within  the  RNLAF,  and  lead  to  more  organisational 
benefits  from  this  senior  expertise. 

4.5.8  Aircraft  availability 

This  factor  is  closely  related  to  the  central  mission  statement  of  the  RNLAF  and  its 
main  product  airpower.  Partly  due  to  challenges  of  personnel  availability,  but  also 
due  to  problems  with  parts  management,  logistics,  and  other  reasons,  aircraft 
availability  itself  has  become  a  challenge  for  the  RNLAF. 

Although  it  is  important  to  consider  which  problems  are  central  in  an  organisation, 
since  training  and  training  solutions  can  be  part  of  the  answer  to  a  problem,  this 
does  not  seem  to  be  the  case  for  the  possible  use  of  IM  AT  tools  in  the  RNLAF. 
Therefore,  aircraft  availability  will  not  be  taken  into  account  further. 

4.5.9  Reorganisation  and  budget  cuts 

These  two  factors  are  mentioned  here  because  it  is  important  to  realise  that  the 
RNLAF  faces,  together  with  other  defence  organisations  in  the  Netherlands, 
reorganisations  and  budget  cuts.  Reorganisations  take  place  through  the 
redevelopment  of  technical  training  (HI VO,  ATTP  and  MTTP)  as  well  as  in  the  F- 
16  maintenance  reorganisation  at  the  airbases.  As  a  consequence  of  budget  cuts, 
serious  reductions  will  have  to  be  made  both  in  personnel  and  material.  These 
factors,  however,  are  very  general  and  will  not  be  taken  into  account  further  while 
thinking  about  IMAT  solutions  for  the  RNLAF. 

4.5.10  Summary 

In  the  previous  section,  these  organisational  factors  were  described: 

•  Airpower:  the  main  product  of  the  RNLAF. 

•  Defence  Organisation:  an  organisation  characteristic. 

•  Aircraft  Maintenance:  the  largest  technical  (training)  domain  in  the  RNLAF 

•  Flight  Safety/  JAR  &  LE:  international  standards  concerning  flight  safety  for 
aircraft  maintenance  and  training. 

•  IETM  Implementation:  an  important  development  in  aircraft  maintenance  and 
training. 
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•  Availability  of  personnel:  there  is  a  shortage  in  personnel,  both  in  training  and 
maintenance  (also  due  to  budget  cuts). 

•  Aircraft  availability:  a  factor  closely  related  to  the  RNLAF’s  main  product. 

•  Reorganisation  and  budget  cuts:  extensive  reorganisations  and  budgets  are 
foreseen  in  the  RNLAF. 

These  organisational  factors  are  studied  further,  together  with  educational  and 
technical  factors,  through  the  development  of  factor  models  in  the  next  chapter. 


4.6  General  Summary 

In  this  chapter  an  inventory  was  made  of  relevant  ETO  factors  for  the  possible  use 
of  IMAT  tools  in  the  RNLAF.  To  study  the  relevance  and  relationships  between 
these  and  their  influence  on  the  technical  training  process  in  the  RNLAF  with  the 
possible  use  of  IMAT  tools,  factor  models  are  developed  in  the  next  chapter.  Factor 
models  present  an  overview  of  how  ETO  factors  interact  and  relate  to  each  other 
and  influence  the  technical  training  process  with  use  of  IMAT  tools  in  the  RNLAF. 
These  models  will  help  in  determining  the  relative  importance  of  the  factors 
described  in  this  chapter,  related  to  the  possible  use  of  IMAT  solutions  in  the 
RNLAF. 
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5.  IMAT  solutions  for  the  RNLAF 


This  chapter  summarizes  the  second  part  of  the  factor  study,  which  consists  of  the 
development  of  factor  models.  Models  are  developed  to  get  a  clear  view  of  the 
relationships  between  relevant  factors  and  their  influence  on  the  technical  training 
process  with  use  of  IMAT  solutions  in  the  RNLAF.  Two  types  of  factor  models 
have  been  developed  and  described:  a  process  model  and  requirement  models.  The 
results  of  expert  interviews  conducted  to  validate  the  factors  and  factor  models  are 
also  described.  See  for  a  detailed  description  of  the  model  development:  Jacobs, 
van  der  Hulst,  Verstegen  &  Janssen  (2003). 


5.1  Introduction 

A  next  step  in  the  study  of  factors  is  the  development  of  one  or  more  models.  The 
purpose  of  models  in  general  is  to  simplify  reality  in  order  to  conveniently  study 
this.  As  such,  a  model  can  be  seen  as  a  means  to  structure  knowledge  of  a  specific 
part  of  this  complex  reality.  A  model  can  be  used  to  describe  and  study  the  reality, 
but  also  to  predict  the  effects  of  a  certain  change  or  choice  in  this  reality. 


5.2  Why  modeling? 

In  this  research,  modeling  is  used  in  the  first  place  to  create  a  clear  view  how  ETO 
factors  and  relationships  between  these  factors  influence  the  possible  use  of  IMAT 
solutions  in  the  RNLAF  context.  For  this  purpose  process  models  are  developed.  In 
the  second  place,  modeling  is  used  as  a  method  to  define  and  study  the  type  of 
requirements  and  requirement  relationships  for  IMAT  on  one  hand  and  for  the 
RNLAF  on  the  other.  For  this  purpose  so-called  requirement  models  are  developed. 
The  construction  of  these  models  gives  a  clear  view  on  the  type  of  requirements, 
their  relationships,  and  a  method  for  a  more  detailed  study  concerning  relevant 
factors,  requirements  and  requirement  relationships.  This  can  also  lead  towards  a 
more  detailed  advice  (as  described  in  the  final  chapter)  on  the  possible  use  of 
IMAT  solutions  in  the  RNLAF. 

5.2.1  A  Systems  approach 

The  approach  taken  on  modeling  in  this  research  follows  a  systems  approach 
(Kramer  &  De  Smit,  1991). 

Kramer  and  De  Smit  (1991)  give  the  following  definition  of  a  systems  approach: 

1 .  Identify  the  whole,  of  which  the  subject  of  the  study  is  part, 

2.  Explain  the  behaviour  or  attributes  of  the  whole. 
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3.  Explain  the  behaviour  or  attributes  of  the  subject  to  be  explained,  in  terms  of 
its  role  or  function  within  the  whole. 

In  essence,  this  approach  suggests  that  IMAT  solutions  should  be  analysed  in  the 
environment  they  will  apply  to  and  thus  should  be  studied  as  a  part  of  a  larger 
whole.  For  this  research  this  refers  to  the  educational,  technical  and  organisational 
factors  in  the  RNLAF  context. 

System  Definition 

Kramer  and  De  Smit  give,  in  their  book  on  systems  approach,  the  following 
definition  of  a  system: 

“A  set  of  entities  with  a  set  of  relationships  that  exist  between  these  entities ” 

A  system  can  be  defined,  according  to  Kramer  and  De  Smit,  as: 

1 .  A  set  of  entities  in  the  system:  W. 

2.  A  set  of  entities  in  the  environment:  E. 

3.  A  set  of  relations  between  the  entities  in  the  system:  Rww. 

4.  A  set  of  relations  between  the  entities  in  the  system  and  the  environment:  Rew. 

Entities  are  elements  or  parts  of  a  system.  Sometimes  entities  are  also  referred  to  as 
objects  or  elements.  Entities  are  the  smallest  parts  in  a  system  one  wants  to  take 
into  account  and  assign  attributes  to.  In  a  process  model  these  entities  consist  of 
processes  (as  black  box),  products  (necessary  inputs  and  outputs)  and  attributes  of 
products/  processes. 

According  to  set  theory  this  definition  of  a  system  (S)  can  be  noted  as: 

S  =  <W,  E,  Rww ,  Rew> 

In  essence,  a  system  description  comes  down  to  the  establishing  (and  describing) 
the  four  sets  and  analysing  whether  a  set  of  entities  in  the  system  (W)  shows  a 
relationship.  To  be  able  to  describe  a  system,  one  or  more  models  of  a  system  is/are 
used. 

System  definition  for  this  research 

It  is  important  to  describe  what  is  defined  as  a  system,  to  be  studied  and  modelled, 
in  this  research.  The  system  definition  for  the  (possible)  use  of  IMAT  in  the 
RNLAF  is  stated  as  follows: 


“The  technical  training  process  in  the  RNLAF  with  use  of  IMAT  solutions” 
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5.2.2  Method 

The  results  of  the  factor  study  (described  in  Chapter  4)  have  been  used  as  input  for 
the  development  of  a  process  model  and  requirement  models  to  describe  and  study 
relevant  factors  for  the  possible  uses  of  IMAT  solutions  in  the  RNLAF.  For  all 
ETO  factors  a  certain  level  of  importance  (high,  medium  or  low)  was  defined,  as 
well  as  a  redefinition  for  each  of  them  as  system  or  environment  entity.  Then,  the 
factors  were  used  to  develop  a  process  model  to  see  how  the  ETO  factors  and  their 
relationships  influenced  the  primary  process  of  technical  training  with  use  of 
IMAT  tools  in  the  RNLAF. 


5.3  The  process  model 

This  section  describes  briefly  the  development  and  outcome  of  the  final  process 
model  for  the  technical  training  process  in  the  RNLAF  with  IMAT  solutions.  See 
for  a  detailed  description  of  the  model  development:  Jacobs,  van  der  Hulst, 
Verstegen  &  Janssen  (2003). 

5.3.1  Process  model  development 

The  process  model  was  developed  according  to  a  systems  approach,  and  followed 
these  steps: 

•  First,  a  process  model  of  the  technical  training  process  with  use  of  IMAT 
solutions  (the  system)  was  developed  without  interaction  with  its  environment, 
or  in  other  words  as  if  there  were  no  other  factors  influencing  or  interacting 
with  this  training  system.  In  this  model  a  set  of  system  entities  was  determined 
and  some  ETO  factors,  such  as  aircraft  maintenance  and  aircraft  availability, 
were  placed  as  system  entities  in  this  model. 

•  Second,  a  process  model  with  environment  entities  and  environment 
interaction  was  developed.  Initially,  all  remaining  ETO  factors  were  placed  in 
this  model  as  environment  entities. 

•  Third,  the  process  model  was  finalised  by  determining  relationships  between 
the  system  entities  as  well  as  between  the  system  and  environment  entities. 
During  this  part  of  the  model  development  new  entities  (factors)  were  also 
found.  By  using  decomposition  rules  some  of  the  system  and  environment 
entities  (ETO  factors)  were  eliminated  from  the  final  process  model  to  obtain  a 
process  model  focusing  on  the  primary  technical  training  process  with  use  of 
IMAT  tools  in  the  RNLAF. 
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5.3.2  The  final  process  model 

The  final  model,  of  the  technical  training  process  in  the  RNLAF  with  use  of  the 

IMAT  tools,  is  presented  in  the  next  figure. 

The  symbols,  colours  and  boxes  in  the  model  have  the  following  meaning: 

•  Black  lined  boxes  represent  the  main  processes  in  the  system  as  black  boxes. 

•  The  inputs  and  outputs  for  a  main  process  are  depicted  by  blue  texts. 

•  A  new  and  additional  input  is  also  depicted  as  blue  text,  accompanied  by  a 
blue  input  arrow.  A  new  input  is  only  specified,  in  the  model,  the  first  time  this 
is  required  in  the  process. 

•  The  black  arrows  between  the  main  processes  in  the  system  depict  the 
workflow  and  possible  directions  of  this  workflow  in  the  system. 

•  Attributes  of  processes  or  products  are  only  specified  if  they  are  derived  from 
the  study  of  ETO  factors,  e.g.  Technical  Training  Characteristics  and  are 
presented  in  a  box  with  a  coloured  line  connected  to  a  process/  product  with  a 
black  dotted  line  (without  arrows) 

•  Yellow  boxes  depict  educational  entities ,  and  their  influence  on  the  technical 
training  process. 

•  Red  boxes  depict  technology  entities ,  and  their  influence  on  the  technical 
training  process. 

•  All  the  grey  boxes  depict  organisational  entities ,  and  their  influence  on  the 
technical  training  process. 

•  A  coloured  arrow  between  system  and  environment  entities  depicts  an 
influence.  E.g.  the  environment  entity  personnel  availability  influences  the 
system  aircraft  maintenance  and  technical  training. 

•  Between  the  two  horizontally  placed  intersection  lines,  the  IMAT  workflow  in 
the  technical  training  process  is  depicted.  It  represents  the  working  of  the 
IMAT  tools  as  developed  in  the  European  IMAT  project. 
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Figure  5.1:  Final  model  of  the  technical  training  process  in  the  RNLAF  with  use  of  the 
I  MAT  tools. 


What  is  interesting  is  that  organisational  entities  (derived  from  the  organisational 
factors)  mainly  influence  the  process  of  aircraft  maintenance  and  to  a  lesser  extent 
the  technical  training  process.  Educational  entities  mainly  influence  the  technical 
training  and  authoring  process,  and  technical  entities  mainly  influence  the 
processes  in  the  IMAT  workflow.  Important  entities  are  those  affecting  the  primary 
process  of  developing  training  material  with  IMAT  tools,  such  as  ADL-SCORM, 
Data-Security  and  Classification.  Other  important  entities  are  the  ones  which  affect 
both  training  and  maintenance,  such  as  the  IETM  implementation.  Finally,  those 
entities  that  affect  both  processes  of  technical  training  and  authoring,  such  as  e- 
leaming  and  simulation-based  training,  are  very  important,  since  they  influence  and 
relate  with  many  other  factors  at  the  centre  of  the  technical  training  development 
and  technical  training  process. 
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However,  to  be  able  to  give  a  more  detailed  advice  on  the  possible  use  of  IMAT 
solutions  in  the  RNLAF  it  is  necessary  to  study  further  on  the  requirements  that  can 
be  defined  to  be  able  to  work  with  IMAT,  such  as  reusability  and  reliability,  in  the 
RNLAF.  Therefore,  requirement  models  were  developed.  They  are  described  in  the 
next  sections. 


5.4  Requirement  Model  Development 

To  answer  the  question  which  requirements  IMAT  needs  to  fulfil  to  be  able  to 
work  with  IMAT  in  the  RNLAF,  and  which  requirements  the  RNLAF  needs  to 
fulfil,  requirement  models  could  be  defined  as  follows: 

•  Given  a  certain  ETO  entity: 

•  Describe  the  general  (functional)  requirement  for  this  entity; 

•  Describe  the  requirements  IMAT  needs  to  fulfil  to  be  able  to  work  with 
IMAT  solutions;  and  finally 

•  Describe  the  requirements  the  RNLAF  needs  to  fulfil  to  be  able  to 
implement  IMAT  in  the  RNLAF. 

5.4.1  Requirement  Model  Construction 

The  next  steps  were  followed  for  the  development  of  requirement  models: 

•  Assemble  important  entities; 

•  Determine  the  general  functional  requirement  for  these  entities; 

•  Determine  relationships  between  entities  and  requirements  in  the  aspect- 
systems; 

•  Develop  requirement  models  for  various  aspect  systems. 

The  selection  of  ETO  entities  for  the  development  of  requirement  models  was 
based  upon  the  assessed  level  of  importance.  All  ETO  entities  with  relevance 
“high”  were  used.  General  functional  requirements  were  then  assembled  and 
requirement  models  were  developed.  It  was  interesting  to  discover  that  some  of  the 
general  requirements  (e.g.  an  automatic  changes  and  update  process  in  IMAT)  had 
multiple  occurrences  in  different  requirement  models.  Only  requirement  models  for 
ETO  entities  resulting  in  new  or  unique  requirements  are  thus  presented  in  the  next 
section. 

The  requirement  models  presented  in  the  next  section,  are: 

•  Technical  Training  Characteristics  Requirements  Model 

•  Simulation  Based  Training  Requirements  Model 

•  E-leaming  Requirements  Model 

•  Infrastructure  Requirements  Model 

•  Aviation  Organisation  Requirements  Model 

•  IETM  Implementation  Requirements  Model 

•  Personnel  Availability  Requirements  Model 
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5.4.2  Requirement  Models 

The  next  figure  gives  an  overview  of  the  requirement  model  for  technical  training 
and  technical  training  characteristics.  The  model  gives  an  overview  of  the 
functional  requirements  related  to  these  educational  entities.  Based  upon  the 
functional  requirements,  requirements  for  IMAT  and  the  RNLAF  can  be 
determined.  The  legend  placed  below  this  model  can  also  be  used  for  the 
understanding  of  the  other  requirement  models. 


F undone!  Requirement* 


IMAT  Requirement* 


RNLAF  Requirement* 


TT  CherecterteBc*  |  »  Requires  Requeerrent  not  edressed  by  MAT 

•  •  Require*  mutual  adjustments  |  Requirement  not  edreeeed  by  RNLAF  | 

Figure  5.2:  Technical  Training  Requirements  Model 

Up  to  date  training  material  and  high  quality  of  training  are  the  main  functional 
requirements  for  technical  training  in  the  RNLAF.  For  IMAT,  this  results  in  the 
requirement  to  facilitate  an  automatic  change  and  update  for  fragments  and 
instructional  fragments.  Due  to  the  fact  that  the  current  versions  of  IMAT  tools  do 
not  facilitate  this  process  the  IMAT  requirement  is  coloured  red  to  indicate  that  this 
is  a  requirement  that  should  be  addressed  in  an  IMAT  solution  of  the  future.  The 
blue  lines  between  technology  and  organisation  indicate  that  mutual  adjustments 
are  necessary  when  using  IMAT  solutions  in  network-centric  training  solutions. 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  educational 
entity:  Simulation-Based  Training. 


Functional  Requirements 


IMAT  Requirements 


RNLAF  Requirements 


Figure  5.3:  Simulation-Based  Training  Requirements  Model 
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The  main  functional  requirements  for  Simulation-Based  Training  (SBT),  in  the 
context  of  technical  training,  can  be  described  as  up  to  date  training  material  and 
compliancy  with  standards.  For  electronic  (training)  content  the  most  important 
standard  is  ADL-SCORM.  For  aviation-based  technical  training  the  most  important 
standards  are  JAR  &  LE.  ATTP  and  MTTP  are  important  examples  of  SBT  used 
by  the  RNLAF  for  training  of  technical  specialists.  For  SBT  keeping  training  up  to 
date  is  a  challenge  (see  also  chapter  3).  Most  important  IMAT  requirements  are  an 
automatic  update  process  and  a  connection  with  the  authoring  environment  of 
simulation-based  training  environments.  For  the  RNLAF,  the  most  important 
requirements  are  to  define  which  authoring  tools  should  be  selected  to  develop  a 
single  and  integrated  e-leaming  development  and/  or  authoring  environment.  Also, 
it  is  important  to  define  how  the  organisational  process  of  change  and  update 
management  (version  management,  revision  cycles  and  verification  with  suppliers 
of  technical  data  and  systems,  etc.)  should  be  designed  to  support  the  operational 
chain  of  changes  and  updates  in  an  effective  and  efficient  manner. 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  educational 
entity  e-  learning. 


Figure  5.4:  E-Learning  Requirement  Model 

In  the  area  of  e-leaming  ADL  SCORM  compliancy  is  the  main  functional 
requirement,  resulting  in  a  similar  requirement  for  IMAT.  The  blue  lines  between 
ICT  Policy  and  E-Learning  indicate  that  mutual  adjustments  between  these  two 
entities  are  necessary  to  be  able  to  implement  e-leaming  solutions  successfully. 
Although  the  RNLAF  is  active  in  both  fields  separately,  mutual  adjustments  to 
these  two  policies  are  necessary  and  desirable  (see  for  instance  chapter  3  on 
simulation-based  training). 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  organisational 
entity:  aviation  organisation. 
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Figure  5.5:  Aviation  Organisation  Requirements  Model 
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For  the  RNLAF  as  aviation  organisation  JAR  &  LE  Compliancy  and  Flight  Safety 
are  the  most  important  main  functional  requirements.  Due  to  extensive  use  of 
country  -specific  data,  RNLAF  supplement  handling  (in  an  efficient  and  effective 
manner)  is  the  third  functional  requirement.  This  results  in  the  IMAT  requirements 
of  RNLAF  supplement  handling  and  also  an  automatic  change  and  update  process, 
since  the  most  important  specification  derived  from  the  JAR/  LE  standard  is  the 
requirement  to  conduct  maintenance  and  training  with  up  to  date  source  material. 
Currently,  the  RNLAF  creates  RNLAF  supplements  in  PDF  format.  This  format 
can  be  used  by  the  IMAT  Pfinal  set  of  tools  as  input  for  source  material.  For  the 
RNLAF  as  organisation,  civil  and  military  aviation  standards  (JAR  &  LE)  are  the 
main  organisational  requirements. 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  organisational 
entity  Personnel  Availability. 

Entity  Functional  Requirements  IMAT  Requirement*  RNLAF  Requirements 


Figure  5.6:  Requirements  Model  for  Personnel  Availability 

Due  to  a  scarcity  in  personnel  in  the  technical  maintenance  and  training  field  in  the 
RNLAF,  efficient  and  effective  training  and  maintenance  are  a  challenge.  For 
IMAT  this  results  in  the  requirement  of  efficient  tools  supporting  the  development 
of  technical  training  in  an  efficient  and  effective  manner.  For  the  RNLAF,  the  main 
requirements  are,  especially  in  situations  when  personnel  availability  is  scarce,  to 
provide  instructors,  instructional  developers  and  specialists  who  need  training 
sufficient  technical  and  didactic  support  either  to  develop,  give  or  follow  training. 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  technical 
entity:  Infrastructure. 

Ertty  Functional  Requirements  MAT  Requirements  RNLAF  Requirements 


ICT  Po«cy 

Technology 

OrgansMion 

KLUM 


Figure  5. 7:  Infrastructure  Requirements  Model 


For  the  technical  entity  Infrastructure,  the  main  functional  requirement  can  be 
formulated  as  “KLUIM  Compliancy”.  This  means  that  new  technology  solutions 
such  as  IMAT  should  be  able  to  fulfil  the  demands  made  by  this  confidential 
network.  On  the  other  hand  the  RNLAF  should  carefully  determine  and  adjust  its 
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ICT  policy,  technology  and  organisation  to  enable  the  efficient  and  effective  use  of 
advanced  training  solutions. 

The  next  figure  gives  an  overview  of  the  requirement  model  for  the  organisational 
entity:  (I)ETM  implementation. 


Ertty  Functional  Requirement  IMAT  Requirement  RNLAF  Requirement 


The  main  functional  requirements  that  can  be  derived  from  the  (I)ETM 
implementation  in  the  RNLAF  are  integration  in  training  on  one  hand  and 
integration  in  maintenance  on  the  other.  For  IMAT  this  results  in  the  requirements 
(I)ETM  connection  and  an  automatic  change  and  update  process  (due  to  changes 
and  updates  in  (I)ETMs).  For  the  RNLAF  the  main  requirements  are  access  to 
ETM  fragments  and  access  to  the  (I)ETM  database.  The  last  two  requirements  are 
crucial,  in  case  the  RNLAF  wants  to  create  training  material  based  upon  (I)ETM 
source  material.  At  this  moment,  the  RNLAF  holds  no  access  to  the  source  material 
outside  the  presentation  modules  provided  by  the  suppliers. 

5.4.3  Requirements  Summary 

The  requirement  models  have  resulted  in  the  next  requirements  for  IMAT: 

1 .  Automatic  change  and  update  process :  this  functional  requirement  was 
presented  5  times  as  requirement  for  IMAT  in  various  requirement  models. 

2.  A  connection  with  (SBT  and  e-learning )  authoring  environments:  this 
requirement  is  not  only  a  requirement  for  IMAT  but  also  for  the  RNLAF, 
implying  an  organisational  choice  (see  also  the  next  list  of  requirements  for  the 
RNLAF). 

3.  ADL  SCORM  Compliancy:  through  the  development  of  a  SCORM  XML 
module  in  IMAT,  IMAT  content  can  be  delivered  in  an  ADL  SCORM 
compliant  manner. 

4.  (I)ETM  connection:  access  to  (I)ETM  content  is  essential,  and  implies  that 
IMAT  should  be  able  to  handle  or  deliver  content  in  data  formats  compliant 
with  (I)ETM  data  formats. 

5.  RNLAF  supplement  handling :  in  case  technical  training  should  also  include 
RNLAF  supplements,  IMAT  should  be  able  to  handle  the  data  formats  of 
RNLAF  supplements.  This  is  the  case  at  this  moment  (IMAT  can  handle  PDF 
documents). 
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6.  KLUIM  Compliancy:  IMAT  should  fit  into  the  RNLAF  ICT  policy  and 

infrastructure  (KLUIM)  to  be  able  to  implement  and  use  IMAT  solutions  in  the 
RNLAF. 

The  requirements  1  and  3  directly  result  in  changes  or  adjustments  of  the  IMAT 

concept.  Requirement  2,  4  and  5  could  result  in  changes  in  the  concept  of  IMAT 

tools  depending  on  necessary  data-formats.  Requirement  6  can  be  seen  both  as 

IMAT  requirement  and  organisational  requirement  for  the  RNLAF. 

The  requirement  models  have  resulted  in  the  next  requirements  for  the  RNLAF: 

1 .  Change  &  Update  Management:  by  this  is  meant  that  the  RNLAF  needs  to 
assist  and  set  up  a  change  and  update  management  system  from  an 
organisational  point  of  view  similar  to  the  current  (I)ETM  change  and  update 
process,  enabling  efficient  and  effective  distribution  of  source  and  training 
material. 

2.  Integrated  ICT  Policy  and  E-learning  Policy:  although  the  RNLAF  works 
actively  on  both  policies  separately,  it  will  become  very  important  to  adjust  and 
mutually  adapt  these  two  policies  to  create  a  technical  and  organisational  sound 
solution  for  advanced  training  solutions  now  and  in  the  nearby  future. 

3.  Integrated  ICT  Policy ,  Technology  and  Organisation:  by  centralising  those 
parts  of  the  organisation  responsible  for  ICT  policy  and  technology  choices, 
(standardisation,  tools,  databases,  etc)  demands  on  this  matter  for  IMAT 
solutions  can  be  set  in  a  clear  manner,  consistent  with  current  and  future  ICT 
developments  in  the  RNLAF. 

4.  JAR  &  LE  Compliancy:  this  requirement  will  become  increasingly  important 
for  the  RNLAF  as  aviation  organisation.  However,  it  is  good  to  realise  that  the 
RNLAF  holds  an  authorising  position  in  this  standardisation  community, 
enabling  the  definition  of  an  own  set  of  standards. 

5.  Access  to  (I)ETM  content:  currently  the  RNLAF  has  no  proper  access  to 
(I)ETM  data  in  the  databases.  However,  full  access  to  (I)ETM  data  is  crucial, 
in  case  the  RNLAF  wants  to  develop  up-to  date  training  material  for  technical 
training. 

6.  Provide  didactic  &  technical  support  for  instructional  developers :  a  certain 
amount  of  training  development  will  always  have  to  be  done  within  the 
RNLAF.  Didactic  and  technical  support  for  instructors  is  very  important  to 
enhance  the  quality  and  standardisation  of  training  material. 


5.5  Model  Validation 

Model  validation  can  be  done  for  different  reasons  or  in  other  words  answer 
different  criteria.  Examples  of  criteria  for  which  model  validation  can  be  set  up, 
are: 

•  Model  results:  is  model  construction  useful?  Does  a  model  generate  the  results 
as  anticipated? 


TNO  report 


76 


•  Model  usefulness  and  usability:  is  the  model  valid  with  respect  to  usefulness 
and  usability? 

•  Plausibility:  is  a  model  plausible  in  the  way  it  explains  relationships  and/or  the 
model  behaviour? 

•  Model  completeness :  is  a  model  complete?  Are  all  entities  present  in  a  model? 

Is  the  behaviour  of  all  entities  in  a  model  sufficiently  explained?  What  is  the 
relative  importance  of  entities  in  a  model? 

•  Model  relevance:  is  a  model  relevant  for  the  type  of  research? 

•  Model  correctness :  is  a  model  correct?  This  means,  are  all  type  of  entities  and 
relationships  in  a  model  correctly  described? 

•  Transparency:  is  a  model  transparent  (or  easily  readable?) 

•  Model  reliability:  Is  a  model  reliable? 

•  Model  construction :  is  a  model  valid  from  a  construction  point  of  view? 

Due  to  the  fact  that  models  are  used  in  this  research  as  method  to  obtain  an  answer 
on  the  possible  use  of  IMAT  solutions  related  to  important  ETO  factors  in  the 
RNLAF,  model  validation  has  been  set  up  to  with  a  strong  focus  on  model 
completeness  (are  factors  missing?).  For  this  purpose  two  expert  interviews  were 
conducted  (to  validate  the  process  models)  and  an  IMAT  workshop  (to  validate 
mainly  the  requirement  models). 

Model  correctness  and  model  construction  are  two  important  criteria  from  a 
construction  point  of  view  and  have  also  been  selected  as  validation  criteria.  For 
this  type  of  model  validation  two  TNO-experts  on  model  construction  have  been 
consulted. 

5.5.1  Expert  Interviews 

This  section  gives  a  summary  of  the  main  results  from  two  interviews  conducted 
with  RNLAF  experts.  Both  experts  were  interviewed  because  of  their  broad  and  in- 
depth  knowledge  of  the  RNLAF,  especially  in  the  field  of  training  and  education, 
and  technology-based  training  solutions.  Both  experts  also  had  substantial 
knowledge  of  IMAT  and  the  IMAT  project. 

Purpose 

The  general  purpose  of  the  interviews  was  determining  whether  certain  factors 
were  missing  in  the  factor  study  and  models,  and  determining  the  usefulness  and 
relative  importance  of  factors  for  model  construction. 

Method 

Given  the  purpose  of  model  validation  by  the  interviewee's  expertise,  it  is  very 
important  to  give  an  interviewee  the  opportunity  to  generate  freely  own  ideas  and 
expertise  on  relevant  factors  without  presenting  the  factors  and  models  as  described 
in  the  previous  sections.  To  support  and  guide  this  process  a  questionnaire  with 
open  questions  and  a  checklist  with  factors  have  been  developed.  The  checklist  was 
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used  to  mark  factors  mentioned  by  the  interviewee  and  to  fill  in  new  factors 
mentioned  by  the  interviewee.  The  questionnaire  was  used  to  ask  a  certain  question 
in  case  an  interviewee  stopped  generating  ideas,  or  if  the  opposite  was  the  case  and 
remained  focused  on  a  single  topic.  See  also:  Appendix  B. 


Results 

The  most  important  conclusions  from  the  interviews  are  presented  here.  They 
concern  interviewees'  agreement/disagreement  with  certain  factors  in  the  factor 
study  and  models  as  well  as  new  factors  mentioned.  See  for  a  more  elaborate 
description  of  the  interview  results:  Appendix  B. 

In  general,  the  interviewees  mentioned  a  large  number  of  relevant  factors 
concerning  the  use  of  IM  AT  in  the  RNLAF  similar  to  the  factors  described  in 
chapter  3.  Both  interviewees  also  mentioned  several  new  and  interesting  factors. 
They  are  listed  in  the  next  table. 

Table  5.1:  New  factors  mentioned  in  interviews 


Interview  1 

Interview  2 

1.1  Effort:  What  does  it  take  to  develop 

training  material  from  technical  manuals, 
with  the  use  of  1MAT? 

2. 1  Effort:  What  does  it  take  to  develop  training 
material  from  technical  manuals,  with  the  use 
Of  IMAT? 

1.2  Costs:  both  time  and  money. 

2.2  Costs:  both  time  and  money. 

1.3  Efficiency:  how  efficient  is  working  with 
an  IMAT  solution? 

2.3  Efficiency:  how  efficient  is  working  with  an 
IMAT  solution? 

- 

2.4  Organisation  culture:  Is  an  important  factor 

2.5  New  developments  in  Technical  Training: 
Technical  training  redevelopment 

Competency  based  training 

Use  of  4CID  model 

1.6  Implementation  plan:  get  commitment  for 
IMAT,  think  about  change  and  innovation 
management. 

2.6  Didactic  support  during  development  (e.g. 
SCO-Generator) 

1.7  Purple  policy^6:  affecting  training  and 

education. 

2.7  TT  design  (including  didactic  support) 

1.8  Reduction  of  personnel:  due  to  budget 
cuts. 

2.8  Reduction  of  personnel. 

1.9  ICT  Policy:  IMAT  should  be  compliant 
with  RNLAF'  ICT  policy  and  standards 
(database  management,  formats,  standards, 
etc) 

2.9  ICT  organisation,  laws  and  rules  concerning 
KLUIM  hold  strong  restrictions  for  the 
implementation  of  e-  learning  and  IMAT 
solutions. 

^  Purple  Policy:  The  Netherlands  Armed  Forces  are  centralising  parts  of  their 
organisations  into  joint  (staff)  organisations,  e.g.  the  central  distribution  of 
personal  gears  and  a  central  staff  for  training  and  education  policy.  This  is 
referred  to  as  “purple  policy”.  The  purpose  of  this  re-organisation  Is  to  enhance 
efficiency  and  effectiveness  in  the  Netherlands  Armed  Forces. 
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Conclusions 

Ad  7.7-7. 3  &  2.1-  2.3:  Effort,  costs  and  efficiency  are  three  factors  that  should  be 
studied  carefully  to  enable  the  RNLAF  to  make  a  well-defined  choice  whether  to 
use  IM  AT  solutions  or  not.  Although  the  IMAT  follow-up  research  is  aimed  at 
defining  an  answer  to  the  possible  use  of  IMAT  solutions  in  the  RNLAF,  it  is  not 
aimed  at  specifying  e.g.  a  return  on  investment  for  IMAT  tools,  or  investigating  the 
financial  consequences  of  development/  procurement  and  use  of  IMAT  tools.  This 
will,  however,  become  an  important  factor  in  case  the  RNLAF  decides  to 
investigate  the  possible  use  of  IMAT  tools  in  the  RNLAF  further,  and  should  be  a 
topic  for  future  research. 

Ad  2.4  and  1. 7:  Although  organisation  culture  and  purple  policy  are  important 
factors  to  take  into  account;  their  relation  with  the  possible  use  of  IMAT  solutions 
in  the  technical  training  process  is  rather  indirect  or  general  and  should  not  be 
included  in  the  process  models. 

Ad  2.5 ,  2.6  and  2.7  New  developments  in  TT,  didactic  support  and  TT  design  are 
important  factors  already  (partly)  addressed  in  the  process  models,  but  will  be 
studied  further  to  see  whether  new  requirements  for  IMAT  solutions  and  the 
RNLAF  can  be  formulated. 

Ad  2.5: Due  to  the  implementation  of  the  4CID  model  in  technical  training  (e.g.  the 
ATTP)  in  the  RNLAF  it  is  interesting  to  find  out  in  what  way  the  instructional 
scenarios  tool  offers  didactic  support  that  matches  or  differs  from  the  4CID  Model. 
Adi. 6:  It  is  important  to  develop  an  implementation  plan  and  think  about  change 
and  innovation  management.  However,  this  factor  will  become  an  important  issue 
when  the  RNLAF  decides  to  implement  IMAT  tools,  and  as  such  would  be  an 
important  requirement  for  the  RNLAF  to  take  into  account. 

Adi. 8:  Although  this  factor  (reduction  of  personnel)  was  identified  in  chapter  3,  it 
was  interesting  to  see  that  both  interviewees  foresaw  a  scarcity  of  personnel  in  the 
nearby  future  due  to  budget  cuts. 

Ad  1.9  and  2.9:  ICT  organisation  and  ICT  Policy  are  important  factors  that  also 
have  been  described  as  requirements  in  the  infrastructure  requirement  model  and  e- 
leaming  requirement  model. 

New  factors  that  came  up  during  the  interviews  were  placed  in  a  process  model  as 
presented  in  figure  4.3.  The  kind  of  relationships  between  the  new  entities,  their 
relevance,  and  level  of  aggregation  were  determined.  Additional  requirement 
models  were  developed  for  those  entities  resulting  in  new  requirements  for  IMAT 
and  the  RNLAF. 

TT  Design  is  an  example  of  a  new  factor  for  which  a  requirement  model  is 
developed,  which  resulted  in  additional  requirements  for  IMAT.  The  next  figure 
gives  an  overview  of  a  requirement  model  for  the  educational  entity  TT  design. 
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Figure  5.9 :  Technical  Training  Design  Requirements  Model 


For  TT  design  the  main  functional  requirements  can  be  described  as  providing 
didactic  and  technical  support  and  knowledge.  For  1MAT  this  results  in 
requirements  for  tools  that  facilitate  organisational  memory  e.g.  in  the  form  of 
annotations  (as  present  in  the  IM  AT  Pfinal  tools).  For  the  RNLAF  the  main 
requirement  is  to  support  and  develop  knowledge  management.  This  is  very 
important  for  an  organisation  as  the  RNLAF  because  of  the  high  level  of  job 
rotations.  IMAT  tools  can  assist  knowledge  management  in  a  practical  and 
concrete  way. 


5.6  Summary 

This  chapter  has  resulted  in  a  clear  overview  of  how  the  collected  ETO  factors 
influence  the  possible  use  of  IMAT  solutions  in  the  technical  training  process  of 
the  RNLAF  through  the  development  of  a  process  model  and  various  requirement 
models.  IMAT  and  RNLAF  requirements  were  set  up.  These  will  be  validated  in 
scenario  assignments  in  an  IMAT  workshop.  See  for  more  information  on  this 
topic,  the  second  IMAT  report,  IMAT  Follow-up  Research  Part  II  (Janssen,  et.  al. 
2004). 
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6.  Conclusions  &  Recommendations 


The  RNLAF  is  a  relatively  small  defence  organisation  with  a  broad  mission.  It  is 
an  organisation  that  possesses  a  wide  range  of  (weapon)  systems  and  related 
technical  functions;  all  operated  with  relatively  few  personnel.  Efficient  and  cost- 
effective  use  of  personnel  training  and  equipment  is  therefore  crucial.  In  this 
context  an  IM  AT  approach  for  the  efficient  reuse  of  the  content  of  technical 
manuals  for  training  purposes  offers  many  benefits.  At  this  moment  the  tools  are 
still  prototype  tools  and  suffer  from  problems  which  make  them  unsuitable  for 
direct  implementation  in  the  RNLAF  organisation. 

Technical  manuals  will,  in  the  future,  be  delivered  in  some  electronic  format.  Just 
as  now,  the  content  of  technical  manuals  will  be  the  basis  for  instructional  material. 
This  will  even  be  required  by  JAR  standards.  So  the  RNLAF  needs  to  think  about 
effective  and  efficient  ways  to  reuse  material  from  technical  manuals  for  training 
purposes,  and  address  issues  such  as: 

•  establishing  search  mechanisms  to  find  appropriate  material  in  the  technical 
manuals; 

•  ensuring  full  access  to  source  materials; 

•  keeping  instructional  material  up-to-date  and  in  accordance  with  updates  of  the 
technical  manuals;  and 

•  sharing  knowledge  about  source  and  instructional  material  between  instructors, 
but  also  between  instructors  and  developers  within  and  outside  the  RNLAF. 

The  IMAT  approach  offers  a  solution  to  these  problems  by  using  ontologies  to 
index  source  material  (from  the  technical  and  didactic  viewpoints)  and  annotation 
mechanisms  to  support  knowledge  management.  The  results  of  the  IMAT  follow¬ 
up  research  show  that  for  complete  implementation  of  the  IMAT  approach 
investments  would  be  required  that  to  be  worthwhile  for  the  RNLAF  will  depend, 
amongst  other  things,  on  the  choice  the  RNLAF  makes  regarding  outsourcing  of 
training  development  and  training. 

In  this  chapter  we  first  describe  the  requirements  that  would  have  to  be  met  for  a 
successful  implementation  of  the  IMAT  approach  in  the  RNLAF.  In  the  last  section 
we  conclude  with  recommendations  for  an  IMAT  roadmap. 

6.1  Requirements  for  the  implementation  of  the  IMAT  approach 

We  have  differentiated  between  requirements  for  the  IMAT  tools  and  requirements 
for  the  RNLAF  organisation.  The  requirements  are  derived  from  the  work  as 
described  in  the  IMAT  Follow-up  Research  Reports,  Part  I  and  II. 
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6.1.1  Requirements  for  the  IMAT  tools 

The  goal  of  the  European  research  project  IMAT  was  to  construct  a  set  of 
prototypes  sufficiently  developed  and  evaluated  to  start  the  construction  of 
commercially  exploitable  tools.  As  a  consequence  the  current  tools  need  to  be 
further  developed.  Below  we  list  requirements  that  are  currently  not  met,  starting 
with  the  most  important  ones: 

•  Implementation  of  an  automatic  change  &  update  mechanism  for  the  IMAT 
database 

This  should  enable  the  reusability  of  previously  indexed  fragments  after  an 
update  of  the  source  material.  In  addition  to  the  version  management  facilities 
in  the  IMAT  Pfinal  tools,  change  and  update  notifications  and  “fragment  or 
text  comparison”  should  become  part  of  future  IMAT  tools. 

•  Implementation  of  an  (semi-)automatic  change  &  update  mechanism  for  the 
instructional  material 

Facilities  to  keep  instructional  material  accurate  and  up  to  date  are  a 
requirement  for  the  successful  application  of  the  IMAT  approach  within  the 
RNLAF.  This  requires  further  development  of  two  aspects  not  currently 
addressed  in  the  prototype  tools: 

-  an  update  mechanism  to  update  the  IMAT  database  without  losing  added 
information,  such  as  meta-data  and  annotations,  and 

-  a  mechanism  to  keep  instructional  material  in  accordance  with  the  updated 
database. 

Regarding  the  second  aspect,  several  solutions  seem  possible.  One  would  be  to 
set  up  a  direct  link  between  the  tools  that  are  used  to  present  instructional 
material  and  the  IMAT  database.  In  this  case,  each  time  that  the  instructional 
material  is  presented,  the  required  fragments  are  retrieved  in  real  time  from  the 
(up-to-date)  IMAT  database.  Another  solution  would  be  to  integrate  an 
authoring  tool  within  the  IMAT  tool  set.  In  this  case  a  dedicated  interface  can 
be  constructed  to  allow  update  of  source  material  to  propagate  automatically  to 
the  instructional  material,  or  to  provide  notifications  at  locations  in  the  lesson 
material  where  a  modified  fragment  is  used  (so  that  the  developer  will  then  be 
able  to  make  the  required  changes  e.g.  by  clicking  a  hyperlink  to  the  source 
material).  From  an  organisational  point  of  view  this  requires  that  the  RNLAF 
will  have  to  select  a  single  or  very  limited  number  of  authoring  and  e-leaming 
platforms. 

•  Ensuring  JAR  compliance 

Important  implications  of  this  standard  concern  the  way  training,  assessment 
and  aviation  maintenance  should  be  set  up  by  the  RNLAF.  Most  important 
JAR  implication  for  the  IMAT  tools  at  present  is  the  prerequisite  to  conduct 
aviation  training  and  maintenance  with  up  to  date  source  material  (see  above). 


TNO  report 


TNO-DV1  2004  A169 


83 


•  Exchangeability  with  other  tools/  systems 

The  current  prototypes  are  based  on  input  in  the  form  of  PDF  documents. 
However,  in  the  future  the  RNLAF  will  use  different  types  of  (I)ETMs,  and 
those  provide  the  input  for  the  IMAT  tools.  Adaptation  of  the  IM  AT  tools  to 
HTML  and  XML  input  should  not  be  difficult,  as  these  formats  are  already 
used  during  the  analysis  process  (and  in  fact  proved  to  be  possible  with  expert 
help  during  the  European  IMAT  project).  The  IMAT  tools  also  need  to 
communicate  with  or  be  integrated  with  the  authoring  systems  and  LMS  that 
the  RNLAF  chooses  to  use. 

•  Ensuring  ADL-SCORM  compliance  and  interaction  with  the  SCO  generator 
This  requires  an  adjustment  in  the  IMAT  concept.  Basically  a  number  of 
incompatibilities  with  respect  to  the  way  stored  materials  are  structured  and 
how  they  are  labeled  by  means  of  meta-data  need  to  be  overcome.  The  easiest 
and  cheapest  solution  is  to  concentrate  only  on  a  lower  level  connection 
between  IMAT  and  the  SCO  generator.  This  solution  is  based  upon  the 
application  of  IMAT  fragments  as  SCORM  Assets.  The  only  conversion  that 
needs  to  take  place  is  from  IMAT  meta-data  to  SCORM  meta-data,  which  can 
be  performed  manually  in  the  SCO  generator  in  the  meta-data  editor.  The  more 
complex  and  expensive  solution  is  to  provide  also  a  higher  level  connection. 
This  solution  is  based  upon  the  application  of  IMAT  fragments  as  SCOs.  An 
extra  IMAT  module  should  be  created  to  convert  the  structure  of  the  IMAT 
fragments  into  the  SCORM  structure  of  SCOs,  and  if  desired,  also  to  convert 
(partly  automatically)  IMAT  meta-data  to  SCORM  meta-data.  The  data  model 
of  SCORM,  however,  is  not  able  to  facilitate  in  the  didactical  richness  of  the 
IMAT  meta-data.  It  is  suggested  to  use  the  technique  of  profiling,  in  order  to 
mediate  between  the  two  data  models  (See  Veerman,  et.  al.,  2002). 

•  Improve  reliability  of  the  tools 

The  prototype  tools  still  have  a  number  of  problems.  This  is  most  clear  in  the 
most  complex  tool:  the  document  analysis  tool.  When  we  tried  to  fragment  and 
index  the  new  input  document  we  discovered,  for  example,  that  some  parts  of 
the  document  were  not  indexed  and  that  the  number  of  fragments  sometimes 
varied  on  different  computers.  Similar  problems  were  also  reported  in  usability 
studies  during  the  European  IMAT  project.  The  tools  must  support  the 
fragmentation  and  retrieval  of  fragments  in  a  reliable  manner.  In  this,  one 
should  focus  particularly  on  the  database  and  document  analysis  tool, 
especially  given  the  large  amounts  of  data  in  technical  domains. 

•  Improving  the  usability'  of  the  IMAT  tools: 

The  perceived  complexity  of  the  tools  is  higher  than  necessary.  This  seems  to 
be  due  tot  the  fact  that  the  IMAT  concept  is  always  explained  in  relation  to  the 
ontologies  that  underlie  it.  For  non-expert  users  this  is  not  necessary. 
Furthermore,  using  the  IMAT  tools  should  be  made  more  intuitive. 

Based  on  the  requirements  as  described  above,  the  next  figure  presents  an 
overview  of  an  IMAT  concept  for  the  future.  The  transparent  boxes  in  this 
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model  represent  processes,  as  a  black  box.  The  blue  texts  represent  the 
necessary  in-  and  outputs.  A  new  input  is  only  specified  the  first  time  this  is 
required  in  the  process.  Between  the  two  dotted  horizontal  lines,  the  IMAT 
process  flow  is  depicted.  One  of  the  lines  is  coloured  red  to  indicate  that  this 
future  IMAT  solution  is  an  extension  of  the  IMAT  concept  as  defined  in  the 
European  IMAT  project.  Also,  the  red  texts  mark  inputs  or  processes  that  are 
new  compared  to  the  IMAT  concept  as  developed  in  the  European  IMAT 
project. 


Figure  6.1:  Model  of  a  future  IMAT  solution  embedded  in  the  technical  training  process 
of  the  RNLAF. 
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6.1.2  Requirements  for  the  RNLAF  Organisation 

The  successful  implementation  of  the  IMAT  approach  also  requires  changes  in  the 

RNLAF  organisation.  These  are  listed  below: 

•  Implementation  of  a  change  &  update  management  process  for  IMA  T. 

The  RNLAF  needs  to  assist  and  set  up  a  change  and  update  management 
process  that  enables  efficient  and  effective  distribution  of  source  material 
(technical  manuals),  IMAT  databases  with  indexed  fragments  and  training 
material,  similar  to  the  current  change  and  update  management  of  technical 
manuals  and  (I)ETMs. 

•  Integration  with  IETM  implementation. 

It  is  very  important  for  the  RNLAF  to  anticipate  on  the  arrival  of  (I)ETMs  in 
their  organisation  and  to  define  exactly  what  kind  of  maintenance  and  training 
solution  they  will  (and  will  have  to)  provide  for  the  RNLAF  organisation.  If  an 
IMAT  approach  is  applied  then  it  should  be  possible  to  use  the  (I)ETM 
material  as  input  for  the  IMAT  tools.  This  implies  that  the  RNLAF  should  have 
substantial  knowledge  on  IMAT  solutions,  IETMs,  and  developments  in  this 
area.  Establishing  a  contract  relationship  with  IETM  suppliers  or  setting  up 
joint  partnerships  with  other  countries  and  organisations  using  the  same 
systems  will  make  it  possible  to  have  a  say  in  in  the  specification,  development 
and  delivery  of  (I)ETMs. 

•  Ensuring  access  to  electronic  content. 

A  more  specific  aspect  is  the  access  to  electronic  content:  e.g.  IETM  databases 
or  databases  of  advanced  training  solutions  provided  by  external  suppliers.  At 
this  moment  it  is  probable  that  the  RNLAF  will  in  many  cases  not  have  access 
to  source  material,  e.g.  in  (I)ETMs.  This  makes  the  reuse  of  source  material 
extremely  difficult:  not  only  in  the  IMAT  approach,  but  also  in  other  advanced 
training  solutions  such  as  the  ATTP. 

•  Standardisation  of  electronic  content. 

This  is  an  important  means  to  gain  real  exchangeability,  reusability  and  easier 
access  to  electronic  content.  See  also  the  IMAT  requirements  described  in  the 
previous  section. 

•  Ensuring  an  integration  of  ICT-  and  E-learning  policy. 

Mutual  adjustments  between  these  two  policies  are  necessary  to  enable 
successfully  large-scale  implementations  of  advanced  training  solution  today 
and  in  the  (nearby)  future.  These  policies  also  need  to  be  aligned  and  further 
developed  in  order  to  decide  if  and  how  the  IMAT  approach  is  going  to  be 
applied.  They  also  determine  RNLAF  demands  for  IMAT  tools. 
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•  Provide  didactic  &  technical  support  to  instructional  developers. 

It  is  likely  that  a  certain  amount  of  training  development  will  always  have  to  be 
done  within  the  RNLAF.  Didactic  and  technical  support  for  training  developers 
is  important  to  enhance  the  quality  and  standardisation  of  training  material.  An 
example  of  how  such  support  can  be  set  up  is  the  Instructional  Scenarios  Tool 
in  IMAT. 

•  Focus  on  modular  competence-based  training 

IMAT  can  serve  the  development  of  different  kinds  of  instructional  material  in 
different  ways.  It  may  be  used  for  a  centralised  development  of  very  basic 
instructional  material,  but  it  may  also  serve  to  support  training  on  the  job 
where  technicians  can  prepare  themselves  ’just-in-time’  for  a  complex  or  highly 
specialised  task.  IMAT  will  then  help  to  find  the  right  material  rather  than 
serve  as  a  set  of  tools  for  the  construction  of  instructional  material. 

6.2  Recommendations 

Based  upon  the  requirements  as  described  in  the  previous  sections,  the  next  IMAT 
roadmap  can  be  defined  for  the  RNLAF  in  case  the  RNLAF  decides  to  develop  and 
implement  an  operational  set  of  IMAT  tools: 

•  Make  IMAT  solutions  and  knowledge  known  to  other  parts  in  the  RNALF 
and/or  other  parts  of  the  Netherlands  Armed  Forces. 

•  Use  the  lessons  learned  in  the  IMAT  project  and  IMAT  follow  up  research  to 
set  up  the  specifications  for  an  operational  set  of  IMAT  tools. 

•  Define  how  an  IMAT  solution  should  fit  into  the  ICT  policy,  e-leaming  policy 
and  ICT  standards  adapted  by  the  RNLAF. 

•  Adjust  ICT  policy  and  educational  policy  to  be  able  to  benefit  fully  from 
advanced  training  solutions  such  as  the  IMAT  approach. 

•  Select  an  authoring  environment  or  e-leaming  platform  to  create  a  closed 
training  (development)  environment  in  which  a  change  and  update  mechanism 
can  be  provided  using  an  IMAT  approach. 

•  Set  up  an  implementation  plan  (with  a  focus  on  change  and  innovation 
management)  for  those  parts  of  the  RNLAF  organisation,  which  are  going  to 
work  with  IMAT  tools. 

•  Verify  how  the  purple  policy  affects  or  can  be  used  for  the  implementation  of 
the  IMAT  approach  in  the  RNLAF.  This  can  be  done  together  with  other 
partners  in  the  Netherlands  Armed  Forces  such  as  the  Royal  Netherlands  Navy, 
which  holds  substantial  knowledge  of  e.g.  (I)ETMs. 

Even  when  the  IMAT  tools  are  not  further  developed,  the  RNLAF  can  make  use  of 
concepts  that  were  developed  within  the  IMAT  projects  and  the  lessons  learned. 
The  Instructional  Scenarios  Tool  (for  the  construction  of  instructional  scenarios), 
for  example,  can  also  be  used  outside  the  IMAT  context.  The  ontologies  and 
annotation  framework  can  be  used  in  other  tools,  and  they  can  serve  as  an 
inspiration  for  demands  or  expectations  from  IETM  facilities.  Experience  with  the 
IMAT  approach  and  prototype  tools  can  serve  as  a  stimulus  to  think  about  different 
ways  of  organising  the  development  and  use  of  instructional  material.  If  further 


TNO  report 


87 


TNO-DV1  2004  A 169 


development  of  the  prototype  tools  will  take  place,  the  three  IMAT  follow-up 
Research  Reports  (Part  I,  II  and  the  report  on  an  ADL  SCORM  Compliant  IMAT 
System)  should  form  the  basis  for  a  needs  statement. 
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Appendix  A  Interview  Reports 

A.l  Interview  method 

Purpose 

•  Validation  of  factor  model 

•  Discover  and  discuss  important  and  relevant  developments  concerning  the  use 
of  IM AT  solutions  in  the  RNLAF 

•  Add  missing  factors  to  the  model 

•  Delete  ‘‘false”  factors  from  the  model 

Introduction 

TNO  is  currently  seeking  to  define  a  set  of  relevant  factors  concerning  the  use  of 
IMAT  solutions  in  the  RNLAF.  The  purpose  of  this  ‘factor  analysis’  is  the 
definition  of  relevant  factors  in  the  RNLAF  resulting  in  a  factor  model.  This  model 
can  guide  the  development  of  relevant  scenarios  concerning  the  various  use  of 
IMAT  solutions  in  the  RNLAF.  Interviews  will  take  place  with  senior  specialists 
with  broad  knowledge  of  the  RNLAF,  especially  in  the  field  of  (technical)  training 
and  education  and  preferably  knowledge  of  IMAT  solutions.  If  the  latter 
knowledge  is  not  present,  a  presentation  of  IMAT  prior  to  the  interview  can  solve 
this  issue.  The  purpose  of  the  interviews  is  validation  of  the  factor  model,  which 
has  been  designed  as  a  result  of  desktop  research  including  a  literature  study, 
project  members’  knowledge  of  the  RNLAF  and  their  TNO  projects  for  the 
RNLAF  in  the  field  of  training  and  education  (including  IMAT,  IETM,  COOL  and 
MTTP). 

IMAT  is  a  technology-based  training  (development)  solution.  When  considering 
the  use  of  such  a  solution  it  is  useful  for  an  organisation  to  look  at  this  from  various 
angles.  This  project  proposes  an  educational,  technical  and  organisational  approach 
and  all  interviewee  should  think  about  and  describe  all  the  relevant  factors  for  the 
possible  use  of  IMAT  solutions  in  the  future  for  the  RNLAF  from  these  three 
angles  of  approach. 

Method 

As  mentioned  the  purpose  of  the  interview  is  validation  of  the  factor  model  by  the 
interviewee’s  expertise.  It  is  therefore  important  to  give  interviewees  the 
opportunity  to  generate  and  present  their  own  ideas  and  expertise  freely  on  relevant 
factors  without  presenting  factors  or  the  factor  model  to  them.  To  support  and 
guide  this  process  a  checklist  of  questions  and  factors  has  been  developed  which 
can  be  used  as  checklist  (and  can  be  marked  when  an  interviewee  mentions  a 
certain  factor  or  answers  a  certain  question).  Or  can  be  used  to  ask  a  certain 
question  in  case  an  interviewee  stops  generating  ideas,  or  if  the  opposite  is  the  case, 
focuses  on  a  single  topic. 
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A.2  Questionnaire 

Algemene  opmerkingen: 

•  Vragen  beperken  zich  tot  de  scope  van  de  KLu. 

•  Met  IMAT  bedoelen  we  EMAT  concept. 

Algemene  vragen 

1 .  Wat  speelt  er  bij  de  Klu  op  het  gebied  van  TT? 

-  Noem  de  drie  belangrijkste  ontwikkelingen? 

-  Wat  zijn  de  belangrijkste  uitdagingen  en  knelpunten? 

2.  Wat  moet  er  veranderen  aan  IMAT  om  IMAT  binnen  de  Klu  inzetbaar  te 
maken? 

3.  Wat  moet  er  in  de  organisatie  gebeuren  om  met  IMAT  te  kunnen  werken? 

Educatieve  vragen 

Welke  educatieve  factoren  zijn  van  belang  bij  het  inzetten  van  ondersteunende 
tools  bij  het  omzetten  van  bronmateriaal  uit  technische  handleidingen  in  instructie 
materiaal  (zoals  IMAT)? 

Algemeen 

1 .  Wat  is  de  omvang  en  het  belang  van  technische  opleidingen  in  de  KLu? 

2.  Wat  zijn  typische  kenmerken  en  uitdagingen  van  TT? 

3.  Is  er  naar  verwachting  in  de  toekomst  een  grote  toestroom  van  student  en? 

4.  Hoe  zou  u  de  studenten  binnen  de  Klu  typeren? 

-  Wat  is  hun  achtergrond  en  ervaring? 

-  Heersen  er  grote  verschillen  in  bekwaamheden? 

5.  Hoe  zou  u  de  instructeurs  binnen  de  Klu  typeren? 

-  Wat  is  hun  achtergrond  en  ervaring? 

-  Heersen  er  grote  verschillen  in  bekwaamheden? 

6.  Is  er  een  grote  diversiteit  aan  trainingen  binnen  de  Klu? 

-  Op  wat  voor  vlakken? 

(Bijv.  meer  focus  op  Hands  On  Training,  meer  focus  op  feitenkennis,  leren  van 
complexe  technische  vaardigheden,  procedures  leren,  etc.) 

7.  Zijn  er  belangrijke  verschillen  /overeenkomsten  tussen  de  KLU  en  civiele 
bedrijven  (bv.  luchtvaartindustrie)  in  de  wijze  waarop  zij  TT  inrichten?  En 
andere  Luchtmachten? 

Ontwerp 

8.  Wat  zijn  belangrijke  kenmerken  van  TT  binnen  de  Klu  om  rekening  te  houden 
bij  het  ontwerpen  en  geven  van  TT? 

9.  Waar  zitten  de  grootste  knelpunten  op  het  gebied  van  trainingsontwerp  en 
ontwikkeling  bij  de  Klu?  + 

10.  Streeft  de  Klu  naar  meer  generieke  oplossingen?  (Qua  trainingsontwerp) 
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1 1 .  Streeft  de  Klu  naar  meer  individualistische  trainingsoplossingen?  (adaptieve 
leertrajecten) 

12.  Wil  de  luchtmacht  zijn  specialisten  de  nodige  vrijheid  bieden  in  the  creeren 
van  lesmateriaal?+ 

-  Zo  ja,  op  welke  vlakken? 

-  Welke  specifieke  problemen  (als  die  er  zijn)  worden  daarbij  voorzien? 

1 3.  Is  outsourcen  van  TT  ontwikkeling  een  mogelijk  oplossing  voor  efficiente  TT 
ontwikkeling  bij  de  KLu?  + 

-  zo  ja,  waarom? 

-  zo  nee,  waarom? 

Geavanceerde  trainingsoplossingen 

14.  Wat  zijn  de  ontwikkelingen  in  de  nabije  toekomst  op  het  gebied  van  e- 
leaming? 

15.  Wat  zijn  mogelijke  knelpunten  hierbij? 

16.  In  hoeverre  zijn  standaarden  als  ADL-  SCORM  leidend  bij  e-leaming 
oplossingen? 

1 7.  Zijn  er  belangrijke  ontwikkelingen  op  het  gebied  van  simulatie  gebaseerd 
opleiden  voor  TT  bij  de  KLu? 

-  Wat  zijn  de  huidige  onderkende  knelpunten  hierbij? 

-  En  in  de  toekomst? 

18.  Zijn  er  belangrijke  lessen  te  leren  uit  het  MTTP  /ATTP  traject  aangaande  de 
ontwikkeling  van  instructie  materiaal  voor  een  blended  learning  solution? 

TT  Curriculum 

19.  Wat  zijn  de  belangrijkste  ontwikkelingen  inzake  de  inrichting  van  het  TT 
curriculum?  (training  on  the  job,  blended  learning?) 

20.  Zou  IMAT  hierin  een  bijdrage  kunnen  leveren? 

21 .  Zijn  er  belangrijke  lessen  die  uit  het  HI  VO  project  kunnen  worden  getrokken 
die  ook  kunnen  gelden  voor  IMAT? 

22.  Wordt  IT  gecentraliseerd  (eerste  deel  van  de  opleiding)  bij  de  KMSL? 

23.  Hoe  en  waar  worden  de  zogenaamde  bedrijfsscholen  voor  TT  ingericht? 

24.  Is  outsourcen  van  TT  een  mogelijke  oplossing  voor  het  geven  van  TT  bij  de 
KLu? 

25.  Hoe  belangrijk  is  het  om  met  up  to  date  en  accuraat  lesmateriaal  te  trainen? 
Effectiviteit 

26.  Waar  zitten  de  grootste  knelpunten  op  het  gebied  van  trainingseffectiviteit  bij 
de  Klu?  (niveaus,  verschillen  tussen  school  en  praktijk) 
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Technische  Vragen 

Welke  belangrijke  ontwikkelingen  spelen  er  op  het  gebied  van  techniek  die  van 
belang  kunnen  zijn  voor  de  toepassing  van  IMAT  binnen  de  KLu?  (Bijvoorbeeld: 
netwerk  beveiliging,  infrastructuur,  capaciteit,  etc...  ) 

IMAT 

27.  Wat  zijn  vanuit  een  technisch  perspectief  belangrijke  voorwaarden  voor  IMAT 
achtige  oplossingen  om  te  kunnen  worden  ingezet? 

28.  Is  het  concept  om  ontologieen  in  te  zetten  binnen  IMAT  een  waardevol  idee, 
dat  wellicht  ook  daarbuiten  zou  kunnen  worden  toegepast? 

Organisatie/  Infrastructuur 

29.  Zijn  er  belangrijke  eisen  die  aan  IMAT  worden  gesteld  vanuit  een  technisch 
perspectief? 

-  bijvoorbeeld  ten  aanzien  van  data-  beveiliging 

30.  Is  de  KLu  infrastructuur  voldoende  ingericht  op  nieuwe  technische 
trainingsconcepten  zoals  IMAT  en  e-leaming? 

31.  Hoe  zou  u  het  technology  level  van  de  KLu  omschrijven? 

-  op  individueel  niveau  (computer  expertise) 

-  op  organisatie-  niveau  (infrastructuur) 

KLu -  supplementen 

32.  Hoe  wil  de  KLu  in  de  nabije  toekomst  omgaan  met  de  KLu-  supplementen? 

-  formaat. 

-  changes  &  updates 

33.  Dient  een  IMAT  oplossing  hiermee  rekening  te  houden?  En  deze  te  kunnen 
verwerken? 

Domein 

34.  Hoe  dynamisch  /  veranderlijk  is  het  luchtmacht  domein?  Denk  bij  voorbeeld 
aan  de  vemieuwingen  in  bron  materieel+ 

35.  Zijn  de  domeinen  die  spelen  binnen  de  Klu  onderling  sterk  vergelijkbaar  (denk 
aan  de  domein  complexiteit  zowel  als  wat  er  geleerd  moet  worden  in  het 
domein)? 

36.  Zijn  volgens  u  domeinanalyses  ter  bevordering  van  de  trainingen  de 
investering  waard? 

Organisatiekundige  vragen 

Welke  ontwikkelingen  in  de  KLu  organisatie  hebben  een  belangrijke  invloed  op 
technische  training  en  opleiding  en  de  wijze  van  inrichting  hiervan?  + 

Organisatie 

37.  Welke  organisatiekenmerken  (al  dan  niet  typisch  voor  defensie)  kunnen  van 
belang  zijn  voor  de  inrichting  van  TT  en  IT-  oplossingen  als  IMAT?  + 

38.  Welke  effecten  hebben  de  beschikbaarheid  van  personeel  voor  T&O?  + 
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39.  Zijn  er  knelpunten  aangaande  personeel? 

Wat  voor  soort  personeel? 

-  Instructeurs 
-OTT 

-  Ontwikkelaars 

-  Onderhoudstechnici 

40.  Op  wat  voor  manier  kunnen  efficiente  en  effectieve  training  (ontwikkeling) 
bijdragen  bij  de  inzet  van  schaarse  capaciteit? 

41 .  Welke  kenmerken  en  uitdagingen  voor  de  organisatie  zijn  van  invloed  op  T&O 
en  eisen  die  hieraan  worden  gesteld? 

-  reorganisaties  (van  WOLF  naar  WING) 

-  budget  cut  downs 

42.  Vliegtuiginzetbaarheid:  Op  wat  voor  manier  kunnen  innovatieve 
trainingsoplossingen  bijdragen  aan  een  hogere  vliegtuiginzetbaarheid? 

IETM 

43.  Welke  consequenties  heeft  de  invoering  van  IETMs  voor  TT  binnen  de  KLu?  + 

44.  Is  het  belangrijk  voor  de  KLu  om  overeenkomsten  en  verschillen  tussen  IETMs 
en  IMAT  te  kennen? 

45.  Welke  zaken  zijn  van  belang  om  te  weten  in  relatie  tot  de  ontwikkelingen 
binnen  IMAT  en  IETM  wereld? 

Standaarden 

46.  In  hoeverre  hebben  de  ontwikkelingen  op  het  gebied  van  standaarden  in  de 
luchtvaart  invloed  op  vliegtuigonderhoud  en  training? 

-JAR  146  enJAR  147  + 

47.  In  hoeverre  hebben  zaken  als  flight  safety  implicaties  op  de  wijze  waarop  TT 
wordt  ingericht? 

-  JAR  146  and  JAR  147  implicaties?  + 

KLu  Supplementen 

48.  Moet  een  IMAT  oplossing  rekening  te  houden  met  de  KLu-  supplementen?  En 
deze  aan  kunnen? 

Samemverking  en  internationalisatie/  nationale  aspecten 

49.  Wat  is  de  visie  van  de  Klu  voor  wat  betreft  de  groeiende  intemationale 
cooperatie  tussen  luchtmachten  en  de  invloed  daarvan  op  de  trainingen? 

50.  Spelen  er  locale  aspecten  in  trainingen  die  in  acht  moeten  worden  genomen? 
Denk  bijvoorbeeld  aan  taal,  locale  condities,  verschillen  in  materieel, 
cultuurspecifieke  verschillen  in  organisatie,  houding,  etc.? 
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Organisatiecultuur 

51 .  Staat  de  organisatie  open  voor  veranderingen  in  rollen  en 
verantwoordelijkheden  van  betrokken  partijen  (instructeurs,  specialisten  op  de 
werkvloer,  etc.)? 

52.  Zo  ja,  Erkend  men  dat  dit  veel  investeringen  zal  kosten? 

Beleid 

53.  Hoe  ziet  het  KLu  beleid  ten  aanzien  van  TT  en  geavanceerde 
trainingsconcepten  er  uit? 

-Voor  de  komende  vijf  jaar? 

54.  En  langere  termijn?  Biedt  het  beleid  van  de  Klu  voldoende  speelruimte  voor 
afstemming  tussen  verschillende  onderdelen  van  de  trainingen:  de  werkwijze, 
roostering,  toetsing  en  kwaliteitszorg? 


Extra  Vragen 

•  Zijn  er  andere  factoren  die  van  belang  zijn  voor  de  KLu  en  IMAT  achtige 
oplossingen? 

•  Is  het  belangrijk  dat  de  KLu  op  de  hoogte  is  en  kennis  heeft  van  IMAT  achtige 
ontwikkelingen? 


A.3  Interview  1 
Interview  results 

" Concerning :  relevant  factors  in  the  RNLAF  for  the  future  use  of  IMAT  solutions” 

Interviewee  data 

Name:  Dr.  Ir.  Hilbert  Kuiper 

Function:  RNLAF  Program  leader  T&E  TNO 

Interviewers:  Nieke  Janssen  (TNO  HF)  and  Lesley  Jacobs  (TNO) 

Location:  At  the  interviewee's  office  at  TNO  in  The  Hague 
Time  and  duration:  13.00  -  14.45 
Date:  10-2-2002 

The  interviewee’s  main  responsibility  is  to  manage,  co-ordinate  and  evaluate  TNO 
training  and  education  (T&E)  projects  for  the  RNLAF.  This  also  involves  the  co¬ 
ordination  and  definition  of  future  T&E  research  programmes.  Hilbert  Kuiper  is 
involved  in  programmes  in  the  area  of  e-leaming,  training  developments,  IE  I M 
research,  use  of  e-tools  on  the  job,  IMAT,  tactical  training,  MTDS^7,  etc. 


^  ^  Mission  Training  (through)  Distributed  Simulation 
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General 

Important  issues  in  the  area  ofTT  in  the  RNLAF: 

E-leaming 

blended  learning  (purple  programme) 

training  on  the  job  (but  also  the  issues  concerning  how  the  initial  training  of 
technicians  can  be  improved  so  that  students  obtain  the  knowledge  and  skills  to 
move  faster  and  be  better  prepared  for  their  tasks) 

Bottlenecks  in  the  TT  developments  at  the  RNLAF: 

The  effort  that  it  takes  to  make  manuals  suitable  for  training  (it  would  be  very 
helpful  if  the  manufacturer  provided  task-oriented  manuals), 

Speed  of  electronic  communication  in  the  past;  there  was  often  was  insufficient 
interaction  in  training  material  (CBT,  simulation,  etc.), 

The  infrastructure  of  the  RNLAF  (KLUIM)  has  many  limitations, 

Currently,  there  is  a  lot  of  duplication  between  the  research  done  at  the  various 
defence  organisations  (RNLAF,  RN,  RNA).  For  instance  in  the  area  of  e- 
leaming,  co-ordinating  research  and  projects  on  e-leaming  within  the 
Netherlands  Armed  Forces  will  prevent  duplication  and  lead  to  better 
knowledge  sharing  between  the  defence  organisations  (also,  there  is  a  trend 
foreseen  in  a  purple  training  and  education  research  programme  and  policy) 

What  is  of  interest  in  the  IMAT  concept: 

The  IMAT  project  is  a  good  project  from  a  scientific  point  of  view  and  the  concept 
is  interesting.  The  subject  ‘fragmenting  source  material’  is  an  important  topic  as 
the  future  will  see  many  systems  enhanced  with  electronic  manuals.  In  practice,  the 
instructors  literally  use  the  manuals  to  write  their  lesson  material.  The  IMAT  tools 
offer  opportunities  to  support  this  process. 

What  should  change  on  the  side  of  IMAT  to  make  it  applicable  in  the  RNLAF? 

The  tools  are  complex  to  use  and  knowledge  on  this  matter  within  the  RNLAF  is 
restricted  to  GGW  de  PEEL  (a  participant  in  the  IMAT  project).  It  is  therefore 
crucial  to  know  how  to  translate  the  results  of  the  IMAT  project  into  practice  and 
transfer  the  knowledge  and  lessons  learned  to  other  parts  of  the  RNLAF 
organisation.  The  real  question  however  is  how  to  do  this?  The  first  step  is 
probably  to  make  IMAT  knowledge  more  widely  available  in  the  RNLAF.  When 
this  is  done,  an  important  next  step  is  to  define  a  user  case  and  start  thinking  of 
how  such  tools  can  be  implemented  in  the  organisation  (implementation,  change 
and  innovation  management) 

What  should  change  on  the  side  of  the  organisation  to  make  IMA  T  applicable  in 
the  RNLAF? 

A  first  step  for  a  successful  implementation  is  the  prerequisite  of  commitment 
within  the  RNLAF  to  work  with  IMAT  tools. 
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IMAT  provides  a  possible  solution  that  affects  the  training  development 
process  profoundly,  RNLAF  T&E  policy  should  address  these  matters  and 
include  implementation  factors  such  as  commitment. 

The  IMAT  concept  should  not  be  introduced  across  the  organisation  at  once. 

A  test  case  (e.g.  at  GGW  De  Peel)  ideally  in  a  new  domain  with  no  current 
training  material,  could  be  used  as  a  first  implementation  test,  to  explore 
strengths  and  weaknesses,  before  implementing  the  tools  on  a  wider  scale. 

Educational  factors 

TT  Importance 

Within  the  RNLAF  TT  is  probably  the  largest  part  of  training  and  is  critically 
important. 

Relevant  educational  factors  for  the  applicability  of  IMAT: 

Instructors  have  strong  technical  domain  expertise  and  lesser  didactic  expertise.  In 
practice,  however,  the  roles  of  technical  expert,  instructor  and  instructional 
designer  are  integrated  in  the  one  person.  So  an  instructor  needs  didactical  support 
to  transform  the  source  material,  (technical  manuals)  into  instructional  material. 

Design 

Relevant  TT  characteristics  to  consider  in  the  design  ofTT  at  the  RNLAF: 

In  TT,  the  design  of  simulations  and  animations  is  an  important  and  difficult  task. 
The  stakeholders  are  keen  on  this  and  fidelity  requirements  (level  of  realism),  are 
high. 

Generic  training  design/adaptive  solutions: 

A  trend  at  the  RNLAF  is  towards  more  generic  training  solutions.  On  the  other 
hand,  there  also  is  a  trend  towards  more  adaptive  learning  solutions. 

Freedom  for  specialists  in  TT  design: 

For  TT  design  (for  which  the  choice  is  made  to  do  in-house  development  within  the 
RNLAF,  e.g.  CBT  development  or  e-leaming)  the  RNLAF  would  probably  wish  to 
give  instructors  a  certain  level  of  freedom  in  designing  their  own  material. 

However  technical  specialists  or  experts  (SME)  would  need  to  monitor  the 
development  process. 

Bottlenecks  in  TT  design  and  development  at  the  RNLAF: 

A  main  bottleneck  is  high  development  times. 

Stakeholders  in  TT  have  very  high  demands  on  the  quality  of  training  (for 
example  on  the  issue  of  fidelity  in  for  example  simulation-based  training 
methods). 

The  development  process  requires  domain  expertise,  but  the  RNLAF  does  not 
have  the  people  and  expertise  to  do  all  types  of  training  development  jobs 
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themselves.  This  for  example  concerns  didactical  expertise  that  is  required  to 
design  training  solutions  and  skills  required  for  simulation  development. 

The  airforce  experienced  that,  with  outsourcing  of  training  development,  the 
risk  is  high  that  they  do  not  get  what  they  want. 

Outsourcing  ofTT  development: 

Considering  the  lack  of  expertise  within  the  organisation,  outsourcing  TT 
development  can  be  effective.  However,  it  requires  close  involvement  of  the 
airforce  to  ensure  that  what  is  done  is  what  has  to  be  done.  For  example,  when 
outsourcing  development  of  advanced  CBT,  the  management  and  feedback  process 
should  be  handled  by  the  RNLAF. 

Advanced  training  solutions 

The  airforce  aims  to  take  advantage  of  the  opportunities  created  by  developments 
in  the  area  of  electronic  learning  (such  as  e-leaming  and  blended  learning).  The 
main  goal  of  these  new  training  strategies  is  to  shorten  the  initial  training  time  at 
school.  An  important  development  in  the  area  of  e-leaming  is  the  RNLAF’s 
adaptation  of  ADL-SCORM. 

Reusability  of  material/ADL  SCORM 

The  reusability  of  material  is  important  for  reasons  primarily  of  cost-effectiveness. 
Co-operation  between  countries  will  also  increase  and  require  greater 
exchangeability  of  material  between  countries. 

Important  developments  in  the  area  of  advanced  training  solutions 

An  important  development  is  JSF  and  the  TT  that  will  be  involved,  based  upon 
class  5  IETMS  with  connected  databases  for  maintenance,  logistics,  e-leaming 
and  EPSS  kind  of  solutions 

MTDS  is  another  important  development  concerning  the  connection  of 
multiple  simulator  trainers  for  pilots  (live  flying  with  multiple  pilots  in 
connected  training  simulators) 

Lessons  learned  from  A  TTP/MTTP: 

•  The  ATTP  product  is  not  what  the  RNLAF  wanted.  The  lesson  learned  is  to 
keep  in  close  involvement  wit  the  developer 

•  Fidelity  remains  an  important  topic  in  simulation-based  training  (stakeholders 
want  a  high  degree  of  fidelity,  even  when  research  indicates  this  is  not 
necessary  for  a  certain  part  (or  function)  of  the  training). 
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TT  Curriculum 

Relevant  developments  considering  the  definition  of  the  TT  curriculum : 

Electronic  manuals  and  training  material  developed  from  them  allow  one  to  use 
several  sources  of  information,  to  provide  information  just  in  time,  e.g.  to  look  for 
examples  in  a  central  database  (accessible  via  the  network  allowing  centralised 
maintenance). 

Centralisation  ofTT: 

The  initial  part  of  TT  will  be  centralised  at  the  KMSL  and  the  rest  of  the  training 
will  be  given  at  a  ‘leerbedrijf  (business  school). 

Outsourcing  of  training 

The  new  electronic  means  of  communication  will  increasingly  provide  support  for 
training  at  the  workplace.  Outsourcing  of  training  will  be  interesting  only  for 
specific  parts  of  the  training,  for  instance  specific  parts  of  initial  training  (prepared 
in  another  country  that  has  the  same  weapon  system). 

Up  to  date  training  material: 

Up  to  date  training  material  is  crucial  in  view  of  the  high  speed  of  change  in 
hardware  and  its  related  software. 

Technical  factors 

Technical  developments  of  relevance: 

One  needs  a  central  database,  accessible  from  multiple  (view)points  to  support  the 
new  ways  of  learning  such  as  training  on  the  job/ just  in  time/  just  enough,  IETMs, 
EPSS  and  video  conferencing.  The  right  information  needs  to  be  available  at  the 
right  time. 

Technical  prerequisites  (requirements)  to  be  able  to  work  with  IMAT  in  the 
RNLAF: 

Infrastructure:  KLUEM  currently  prevents  many-advanced  (network)  training 
solutions  from  a  full-scale  implementation. 

Means:  as  an  organisation  the  RNLAF  needs  to  provide  users  of  IMAT  with 
sufficient  resources  to  create  good  instructional  material  (e.g.  computer, 
authoring  tools,  access  to  central  database,  etc.) 

Data  security:  will  be  an  issue  when  working  with  classified  information  in 
technical  manuals  and  training  (not  allowed  on  KLUfM). 

Ontology  Concept: 

It  could  be  useful  to  consider  further  (besides  or  outside  IMAT)  generating 
metadata  for  electronic  content. 
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Organisation/infrastructure 

•  Technical  requirements  for  IMAT:  IMAT  must  respond  to  RNLAF  ICT  policy 
and  be  compliant  with  it  e.g.  the  choices  RNLAF  makes  concerning  database 
management,  electronic  formats  and  ADL  SCORM.  The  IMAT  solution  must 
integrate  in  the  RNLAF  architecture  and  policy. 

•  Data-security  is  (within  the  RNLAF  confidential  network  KLUIM)  only  an 
issue  when  classified  material  is  used. 

Technology  level 

The  RNLAF  is  advanced  in  its  use  of  high  technology  (according  to  the 
interviewee  also  compared  to  other  Dutch  defence  organisations,  e.g.  ADL- 
SCORM).  For  the  students  this  may  depend  on  the  entrance  level. 

RNLAF  Supplements 

It  is  important  that  the  IMAT  concept  conveniently  facilitates  updates  of  RNLAF 
supplements. 

Domain 

The  RNLAF  domains  are  dynamic,  with  regular  updates  e.g.  of  source  material 
once  per  two  months. 

The  RNLAF  domains  are  strongly  similar. 

Domain  analysis: 

An  important  question  is:  what  is  the  cost-benefit  in  time  and  investments  to 
develop  and  maintain  a  domain  ontology. 

Organisational  factors 

Important  characteristics  and  developments  in  the  RNLAF for  the  use  of  IMAT 
solutions 

An  important  question  to  be  addressed  is  how  to  get  commitment  within  the 
RNLAF  for  such  a  training  solution? 

Reusability  is  a  main  topic  for  the  use  and  re-use  of  electronic  content. 

IETM  implementation. 

Consequences  of  budget  cut 
Centralisation  of  training 
Centralisation  of  policy  activities 
Personnel  reductions 

Efficiency 

For  efficiency  reasons  reusability  of  material  is  very  important. 
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Challenges  for  the  organisation 

Increases  emphasis  on  efficiency,  purple  policy  (also  within  the  area  of 
training),  cost  reductions 

Analyse  training  programmes  and  define  doubles  (within  RNLAF  and  other 
Dutch  Defence  organisations) 

Increase  flight  hours: 

New  ways  of  training  could  increase  returns  on  investment  in  training  if  it  leads  to 
more  flight  hours. 

IETM: 

IETMs  support  the  availability  of  information,  probably  in  a  more  effective 
way 

The  IETM  implementation  could  lead  to  a  smaller  part  of  instructor  lead 
training,  and  more  self  directed  learning. 

For  the  RNLAF  it  is  very  important  to  understand  the  exact  difference  between 
IMAT  and  IETMs.  What  offers  IMAT  more  in  the  area  of  training? 

New  ways  of  training: 

A  consequence  of  new  ways  of  training  is  that  the  role  of  an  instructor  changes 
drastically 

International/  local  factors: 

The  RNLAF  increasingly  works  together  with  countries  with  the  same  weapon 
systems  on  matters  as  maintenance  (EPAF  countries)  and  training 
(MTTP/ATTP  example:  partnership  with  Norway  to  set  up  FI 6  maintenance 
training) 

For  instance  for  the  JSF,  choices  can  be  made  whether  the  RNLAF  develops  its 
own  training,  buys  in  training,  or  outsources  certain  parts  of  training  in  other 
countries.  In  essence  this  is  a  matter  of  cost-analysis 
The  most  important  local  factors  are  training  material  and  organisation 

Organisation  culture 

Policymakers  are  progressive  and  open  to  changes  in  roles  and  responsibilities. 
They  however  realise  that  it  will  cost  a  lot  for  people  in  the  field  to  change  and  to 
obtain  commitment  for  a  certain  change/innovation 

Policy 

RNLAF  policy  for  TT  (long  term): 

Just  enough 
Just  in  time 
Blended  learning 
Leerbedrijven 
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The  interviewee  will  see  that  the  interviewers  get  a  copy  of  the  recently  written 
RNLAF  policy  paper  on  future  training  within  the  RNLAF. 

Summary 

New  factors  that  came  up  in  this  interview: 

Effort:  what  does  it  cost  to  develop  training  material  from  technical  manuals 
(with  the  use  of  IM AT)? 

Costs:  both  time  and  money. 

Efficiency:  how  efficient  is  working  with  an  IMAT  solution? 

ICT  Policy:  IMAT  should  be  compliant  with  RNLAF  ICT  policy  and 
standards(database  management,  formats,  standards,  etc) 

Reduction  of  personnel  due  to  budget  cuts. 

Purple  policy7**  affecting  training  and  education 

Implementation  plan:  get  commitment,  think  about  change  and  innovation 
management. 

New  questions  that  came  up  in  this  interview: 

What  is  gained  by  using  IMAT  instead  of  using  IETMs  in  the  area  of  training? 
How  do  you  get  commitment  within  the  RNLAF  for  IMAT? 

How  do  you  get  the  product  that  you  want?  (ATTP  example  but  also  in  IMAT) 


A.4  Interview  2 
Interview  Results 

" Concerning  relevant  factors  in  the  RNLAF  for  the  future  use  of  IMAT  solutions” 
Interviewee  data 

Interviewee:  Dr.  Willem  Hylkema 

Function:  RNLAF  Program  leader  Training  &  Education 

Interviewers:  Nieke  Janssen  (TNO  HF)  and  Lesley  Jacobs  (TNO) 

Interview  setting:  at  the  office  of  the  interviewee  in  the  RNLAF  headquarters  in 
The  Hague 

Time  and  duration  of  interview:  13.15-15.00 
Date:  11-2-2002 

General 

Important  issues  in  the  area  of  TT  in  the  RNLAF: 

The  adaptation  of  ADL  SCORM  for  the  standardisation  of  e-leaming  content. 
TT  in  the  RNLAF  concerns  mostly  the  FI 6/ Apache  domain.  In  general  the 
training  projects  focus  on  a  structural  redevelopment  of  the  training.  Examples 
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in  the  FI 6  domain  are  the  ATTP  and  MTTP  project.  The  Apache  is  a  new 
weapon  system,  and  training  material  is  from  the  USAF. 

Development  costs:  how  to  develop  good  content  cost-effectively? 

Quality  of  simulation  in  technical  training  is  a  serious  issue  (has  been  a 
drawback  in  previous  years). 

In  the  past,  training  development  in  the  RNLAF  has  been  done  mostly  bottom 
up  (instructors  develop  their  own  material).  The  interviewee  indicates  that  this 
is  not  necessarily  a  bad  thing.,  but  from  an  educational  design  point  of  view 
standardisation  and  structured  training  development  such  as  ISD  or  4CID  are 
recently  coming  more  in  focus  in  projects  such  as  HIVO  and  ATTP/MTTP. 

Bottlenecks  in  TT  development  at  the  RNLAF: 

Effort  by  people  (time)  and  financial  resources  to  develop  TT. 

SME  time  (content  expertise)  and  training  development  time. 

What  should  change  on  the  side  of  IMAT  to  make  it  applicable  in  the  RNLAF: 
IMAT  concept  is  good:  how  to  re-use  existing  content  and  information.  This 
could  lead  to  a  cost  reduction  in  TT  development. 

Interesting  questions  to  be  answered: 

How  does  IMAT  relate  to  ADL  SCORM?  Can  IMAT  be  used  to  add  metadata 
to  SCORM  compliant  content? 

To  what  degree  does  the  IMAT  metadata  set  derived  from  the  IMAT 
taxonomies  differ  or  map  with  the  SCORM  metadata  set? 

To  what  extent  is  it  possible  to  connect  IMAT  (fragments)  to  the  SCO- 
Generator?  Is  it  for  instance  possible  to  extract  IMAT  fragments  from  the 
database  and  use  them  in  the  SCO-Generator  to  create  instructional  material,  or 
in  other  words:  could  IMAT  work  as  a  pre-authoring  system  for  the  use  of  the 
SCO-Generator? 

Another  interesting  question  (also  outside  IMAT)  is  the  question  to  which 
extent  metadata  can  be  generated  automatically  for  learning  objects,  instead  of 
manual  creation  of  a  metadata  set.  (the  interviewee  gives  the  example  that  KPN 
currently  estimates  that  80%  of  training  development  time  is  spent  on  the 
(manual)  creation  of  metadata  for  electronic  content). 

What  should  change  on  the  side  of  the  organisation  to  make  IMAT  applicable  in 
the  RNLAF: 

TT  developers  should  be  trained  to  be  able  to  work  with  IMAT  tools. 
Educational  Factors 
General 

Important  developments  in  RNLAF: 

Competence-based  training  and  education. 

Connect  learning  objects  to  competencies. 
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Open  versus  closed  learning  environments. 

Importance  of  TT: 

TT  is  the  core  of  training  and  education  within  the  RNLAF. 

TT  Characteristics: 

Aimed  at  the  training  and  education  of  functions,  and  therefore  on  the  skills  of 
persons  to  perform  a  function. 

Students:  there  is  a  large  variability  in  students,  due  to  a  large  set  of  functions 
and  scales  within  the  RNLAF.  In  general  the  initial  educational  level  of  new 
technicians  in  the  RNLAF  is  declining  . 

Instructors  in  TT  have  an  MBO  degree  in  a  technical  field,  strong  domain 
expertise  (maintenance  experience)  and  additional  education  as  instructor. 
Instructors  are  also  TT  developers  (usually  two  roles  in  one  person). 

Challenges  in  TT: 

In  the  past  TT  has  been  designed  with  a  strong  knowledge-based  focus. 
Therefore  the  transfer  of  training  to  practice  was  a  large  step  (students  would 
have  a  broad  and  depth  knowledge,  but  no  training  in  how  to  use  this 
knowledge  in  practice  during  maintenance). 

At  the  moment  there  is  a  shift  in  the  RNLAF  to  competence-based  training,  JE, 
JIT,  and  learning  tasks.  Focus  during  initial  training  should  lie  more  on  skills, 
both  cognitive  and  psycho-motoric  skills. 

Design 

Relevant  TT  characteristics  to  consider  in  the  design  ofTT  at  the  RNLAF: 

Focus  on  skills  and  design  of  learning  tasks. 

"Leerstofdomein  afbakening”  (defining  subject  limits)  is  an  important  issue. 
Bottlenecks  in  TT  design: 

Capacity  and  effort  of  people  to  create  training  material. 

Moments  of  choice  in  the  instructional  design  process  are  subjective  (there  is 
no  single  good  solution,  there  are  multiple  "good  ways"  in  an  ISD  process). 

TT  design: 

The  RNLAF  aims  at  more  structured  and  standard  support  (as  for  instance 
provided  in  the  SCO-Generator)  during  the  development  of  training  (generic 
solutions,  standardisation) 

The  interviewee  does  not  yet  foresee  more  adaptive  learning  solutions  for 
students.  It  depends  on  the  organisation's  choice  between  a  docent  versus  a 
student  centred  learning  solution.  And  although  from  a  student  perspective 
(certainly  at  school)  a  certain  level  of  adaptivity  is  very  desirable,  (e.g.  self 
study  and  learning  material  aimed  at  various  learning  styles).  The  real  issue  is 
how  to  prevent  grow  of  a  large  variety  of  training  types,  resulting  in  content 
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management  problems  (time  consuming  and  costly).  In  the  end,  the 
interviewee  foresees  a  middle  of  the  road  scenario. 

Design  support: 

There  is  a  need  for  didactic  support  during  the  training  design  and  development 
phase,  and  this  is  besides  IMAT  (in  the  Instructional  Scenarios  Tool)  also 
addressed  in  the  SCO-Generator  project. 

The  RNLAF  thinks  it  is  important  to  provide  instructors  (training  developers) 
with  structured  support  during  TT  design  and  development,  e.g.  through  the 
development  of  EPSS  kind  of  solutions. 

The  RNLAF  wants  to  give  instructors  freedom  to  develop  training  material 
according  to  their  own  expertise  to  a  certain  level  the  (freedom  within  limits). 
However,  freedom  always  will  be  limited  due  to  organisational  choices  e.g.  for 
certain  development  tools,  platform  or  learning  solution  (ATTP  example:  this 
is  not  a  facultative  learning  solution).  The  real  question  is  to  which  extent  an 
organisation  should  allow  individual  freedom  and  at  which  point  the 
organisation  determines  the  development  process  (e.g.  central  content 
management). 

Outsourcing  ofTT  development: 

Yes  and  no.  Due  to  lack  of  in-house  expertise  outsourcing  is  sometimes  the 
only  solution. 

Outsourcing  is  only  possible  at  certain  levels,  and  always  requires  a  large 
involvement  of  RNLAF  SME  expertise  during  the  development  phase. 
Eventually  a  mix  of  in  house  development  and  outsourcing  will  take  place.  The 
IMAT  tools  could  assist  in  the  pre-authoring  process  (to  create  and  retrieve 
fragments)  for  CBT  material,  which  can  be  developed  in  house.  More  complex 
training  solutions,  such  as  simulation,  will  probably  require  outsourcing  due  to 
a  lack  of  in-house  expertise. 

Advanced  training  solutions 

Important  developments  in  the  area  of  advanced  training  solutions: 

E-Learning:  ADL-SCORM 

Challenges:  How  to  set  up  TT  (with  e-leaming)  in  such  way  that  initial  training 
at  the  KMSL  and  training  on  the  job  "match". 

Simulation  Based  Training:  ATTP/MTTP 

Challenges:  How  to  develop  a  coherent  training  system  instead  of  single 
simulation  components?  (In  the  MTTP  project  this  is  the  "showcase  question", 
the  4CID  model  is  used  in  this  project  as  didactic  guidance  in  giving  learning 
means  a  certain  function  in  the  educational  process) 

Lessons  learned  in  ATTP/MTTP  project: 

Pre-phase  is  extremely  important:  think  about  and  define  what  you  want  to 
have. 

There  are  many  opinions  (and  differences  in  expectations)  on  the  format  of 
the  end  product 
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It  is  difficult  to  set  up  good  specifications  (they  are  not  always  clear  to 
developers  and  can  lead  to  differences  in  expectations  of  the  product). 
Tools:  the  development  of  tools  (advanced  training  solutions)  often  is  not 
the  most  difficult  part,  but  providing  and  having  the  didactic  know-how 
how  to  use  them  in  training  is  a  challenge.  Consequently,  the  learning 
means  (such  as  COO)  sometimes  are  not  as  used  in  the  way  as  was  meant 
by  the  instructional  design). 

TT  Curriculum 

Relevant  developments  considering  the  definition  of  the  TT  curriculum: 

Blended  learning  (ILT,  e-leaming,  simulation  based  training,  training  on  the 
job) 

Role  of  IMAT:  IMAT  can  act  as  bridge  between  the  delivery  of  assets  for  basic 
knowledge  and  JIT.  (IMAT  can  help  in  the  easy  disclosure  of  information,  act 
as  a  pre-authoring  system  delivering  ‘semi-finished  goods’:  raw  material 
suitable  for  the  creation  of  training  material). 

Centralisation  ofTT: 

The  initial  part  of  the  IT  will  be  centralised  at  the  KMSL  and  the  rest  of  the 
training  will  be  given  at  a  "leerbedrijf'  (business  schools)  at  the  different  RNLAF 
bases. 

Outsourcing  of  training: 

This  certainly  is  a  possibility,  but  many  scenarios  are  possible  in  the  future 
(examples  of  EPAF  countries  distributing  or  specialising  in  parts  of  FI  6 
maintenance  training  or  in  the  future  JSF  training). 

Up  to  date  training  material: 

Up  to  date  training  material  is  very  important,  but  it  is  better  to  speak  in  terms  of 
valid  training  material  (this  is  not  necessarily  the  same  as  up  to  date  material). 

Valid  training  material  can  also  comprise  a  generic  training  module,  which 
explains  for  instance  the  principles  and  working  of  a  GPS  without  having  the  latest 
documentation  or  exact  version  of  the  GPS  in  a  FI  6. 

Training  Effectiveness: 

The  key  to  efficiency  is  ’’doeltreffendheid/effectiveness”.  Just  enough  training  is  an 
important  feature  here.  Students  need  to  learn  what  they  need  to  know  for 
competent  performance  (rather  than  what  is  nice  to  know). 
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Technical  factors 

Technical  developments  of  relevance: 

IETM  implementation  in  the  RNLAF. 

Technical  requirement's  for  IM AT: 

Availability  of  electronic  content 

Ontology  concept: 

The  interviewee  does  not  see  any  direct  uses  of  the  ontology  concept  outside  of 
IMAT.  The  concept  of  putting  tags  on  fragments  however  is  important  and  IMAT 
provides  solutions  for  that.  However,  the  interviewee  foresees  that  the  RNLAF  will 
start  working  with  a  small  set  of  metadata  derived  from  IMS  (instead  of  a  rich  set 
of  metadata  in  IMAT).  It  is  important  to  determine  whether  the  effort  to  foresee 
fragments  with  metadata  is  worthy  of  the  reusability  factor.  Maybe  it  is  far  more 
interesting  to  think  about  defining  a  set  of  metadata  as  small  as  possible  to 
guarantee  reusability  of  content. 

Organisation/infrastructure 

Data  security  is  not  an  issue  as  long  as  IMAT  operates  within  the  confidential 
RNLAF  infrastructure  (KLUIM).  This  will  be  different  as  soon  as  online  or 
wireless  communication  is  used  as  solution  (interviewee  does  not  foresee  this 
kind  of  communication  necessary  for  the  use  of  IMAT  in  the  RNLAF). 
Infrastructure:  According  to  the  interviewee  KLUIM  is  from  a  technical 
perspective  (bandwidth)  adapted  for  e-leaming  solutions,  but  from  an 
organisational  perspective  many  complications  are  foreseen  due  to 
administrative  rules  (and  the  fact  that  KLUIM  is  maintained  by  a  separate 
organisation:  the  DTO).  Many  solutions,  such  as  ADSL  or  wireless 
communication  are  not  allowed  within  the  confidential  network  of  the  RNLAF. 

Technology  level 

The  RNLAF  is  progressive  in  its  level  of  technology,  certainly  in  areas  where 
people  work  with  advanced  weapon  systems,  advanced  technology  is  commonly 
used  both  in  maintenance  and  training. 

RNLAF  Supplements 

The  interviewee  indicates  that  the  RNLAF  supplements  fall  under  the  jurisdiction 
of  another  department  of  the  RNLAF  (DM)  and  therefore  holds  no  opinion  on  this 
matter. 

Domain 

The  RNLAF  domain  is  dynamic  (e.g.  FI 6),  but  changes  and  updates  are 
predictable. 

The  RNLAF  domains  differ  (e.g.  Apache  is  a  digital  system  and  the  FI 6  is  an 
analogue  system) 
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Domain  analysis  is  in  theory  worthwhile,  in  practice  probably  not.  A  lot  of 
domain  knowledge  is  for  instance  available  with  suppliers  of  a  system  or  other 
airforces.  One  could  for  instance  take  the  approach  to  see  which  knowledge 
can  be  borrowed  or  used  from  other  parties  instead  of  putting  lots  of  effort 
(time  consuming,  high  costs)  in  the  pre-authoring  process  (including  a  domain 
analysis),  of  course  all  within  the  limits  of  safety. 

Organisational  factors 

General 

Training  is  strongly  function-related. 

A  defence  organisation  is  very  structured,  and  has  to  follow  many  rules  and 
laws  (also  outside  the  defence  world,  e.g.  JAR  standards). 

Therefore  (probably)  less  room  is  foreseen  for  open  learning  environments 
(comparison  with  the  Belastingdienst,  tax  authorities  that  follow  the  path  of 
open  learning  environments).  In  an  open  learning  environment  IMAT  tools 
could  for  instance  be  used  as  JIT  tools  by  students.  In  the  RNLAF  context 
(probably  a  closed  learning  environment)  IMAT  tools  will  act  as  pre-authoring 
tools  in  the  training  design  and  development  phase. 

Personnel  availability: 

The  RNLAF  is  a  relatively  small  defence  organisation,  with  a  wide  range  of 
weapon  systems  functions  and  tasks  and  therefore  personnel  availability  is  a 
serious  problem  in  all  functions  (Instructors/Developers,  OTT,  and  Maintenance 
Specialists). 

Consequences  of  budget  cuts: 

More  tasks  then  people  (scarcity  of  personnel)  in  the  RNLAF. 

Personnel  reductions. 

IETM 

The  IETM  implementation  will  probably  lead  to  a  stronger  focus  on  training  on 
the  job  and  EPSS  approaches  in  training,  given  the  prerequisite  that  a  person 
has  the  basic  knowledge  skills  (received  during  initial  training).  The 
interviewee  indicates  that  in  general  he  foresees  a  division  in  specialists 
between  a  large  group  of  specialists  focusing  on  daily  maintenance,  and  a  small 
group  of ’’troubleshooting"  specialists.  EPSS  and  coaching  are  two  elements  of 
support,  which  can  be  provided  during  maintenance. 

In  the  future  (e.g.  JSF)  IETMs  will  be  connected  with  systems  on  board  aircraft 
providing  new  ways  of  assistance  during  maintenance. 

Interviewee  sees  IETM  and  IMAT  as  two  different  solutions  for  two  different 
phases.  IMAT  can  provide  support  during  training  development  and  IETMs 
can  provide  support  during  maintenance  and  maybe  act  as  JIT  solution  on  the 
job. 
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Another  question  is  to  which  extent  IETMs  can  act  as  training  material,  and 
which  steps  writers  of  IETMs  undertake  to  create  "real”  training  material  based 
upon  the  IETM. 

Standards 

JAR  146  and  166  hold  strong  implications  for  TT  for  aircraft  maintenance  (e.g.  up 
to  date  training  material) 

International/  local  factors 

Internationalisation  grows  certainly  between  countries  with  the  same  weapon 
systems  (e.g.  EPAF  countries  conduct  maintenance  (and  other  related  issues 
such  as  logistics)  in  the  same  way,  so  that  during  NATO  missions  technicians 
between  the  airforces  can  work  on  each  others  F16’s) 

Local  factors  play  a  role  to  a  certain  extent,  but  F16systems  maintenance  is 
F16systems  maintenance,  as  much  in  the  Netherlands  as  in  Belgium  or  the 
USA. 

Summary 

New  factors/  issues  that  came  up  in  the  interview: 

New  approaches  in  IT : 

TT  redevelopment 
Competency  based  training 
-  4CID  model 

Didactic  support  during  development  (e.g.  SCO-Generator) 

TT  design  (support) 

ICT  organisation,  laws  and  rules  concerning  KLUIM  hold  strong  restrictions 
for  the  implementation  of  e-  learning  and  IMAT  solutions. 

Reduction  of  personnel 

Organisation  culture 

Effort 

Efficiency 

Costs 

An  asset  is  an  instructional  fragment  (and  not  a  fragment  from  a  technical 
manual) 

New  questions  that  came  up  during  the  interview: 

How  do  you  get  what  you  want  (simulation  based  training,  but  also  in  IMAT)? 
What  does  it  take  (effort)  to  transform  technical  documentation  into  training 
material  with  IMAT? 

Is  the  IMAT  pre-authoring  process  worth  the  costs,  time  and  effort  (in  a  certain 
domain)? 

What  is  the  added  value  of  ontology  development? 

Could  a  smaller  set  of  metadata  do  the  job? 

Can  IMAT  deliver  ADL  SCORM  compliant  content  (or  add  metadata)? 
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